From: Isaku Yamahata <isaku.yamahata@gmail.com>
To: Sean Christopherson <seanjc@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
Marc Zyngier <maz@kernel.org>,
Huacai Chen <chenhuacai@kernel.org>,
Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>,
Anup Patel <anup@brainfault.org>,
Paul Walmsley <paul.walmsley@sifive.com>,
Palmer Dabbelt <palmer@dabbelt.com>,
Albert Ou <aou@eecs.berkeley.edu>,
Christian Borntraeger <borntraeger@linux.ibm.com>,
Janosch Frank <frankja@linux.ibm.com>,
Claudio Imbrenda <imbrenda@linux.ibm.com>,
Matthew Rosato <mjrosato@linux.ibm.com>,
Eric Farman <farman@linux.ibm.com>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
James Morse <james.morse@arm.com>,
Alexandru Elisei <alexandru.elisei@arm.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Oliver Upton <oliver.upton@linux.dev>,
Atish Patra <atishp@atishpatra.org>,
David Hildenbrand <david@redhat.com>,
kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu,
linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org,
linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org,
Isaku Yamahata <isaku.yamahata@intel.com>,
Fabiano Rosas <farosas@linux.ibm.com>,
Michael Ellerman <mpe@ellerman.id.au>,
Chao Gao <chao.gao@intel.com>,
Thomas Gleixner <tglx@linutronix.de>,
Yuan Yao <yuan.yao@intel.com>,
isaku.yamahata@gmail.com
Subject: Re: [PATCH 00/44] KVM: Rework kvm_init() and hardware enabling
Date: Fri, 4 Nov 2022 00:17:49 -0700 [thread overview]
Message-ID: <20221104071749.GC1063309@ls.amr.corp.intel.com> (raw)
In-Reply-To: <20221102231911.3107438-1-seanjc@google.com>
On Wed, Nov 02, 2022 at 11:18:27PM +0000,
Sean Christopherson <seanjc@google.com> wrote:
> Non-x86 folks, please test on hardware when possible. I made a _lot_ of
> mistakes when moving code around. Thankfully, x86 was the trickiest code
> to deal with, and I'm fairly confident that I found all the bugs I
> introduced via testing. But the number of mistakes I made and found on
> x86 makes me more than a bit worried that I screwed something up in other
> arch code.
>
> This is a continuation of Chao's series to do x86 CPU compatibility checks
> during virtualization hardware enabling[1], and of Isaku's series to try
> and clean up the hardware enabling paths so that x86 (Intel specifically)
> can temporarily enable hardware during module initialization without
> causing undue pain for other architectures[2]. It also includes one patch
> from another mini-series from Isaku that provides the less controversial
> patches[3].
>
> The main theme of this series is to kill off kvm_arch_init(),
> kvm_arch_hardware_(un)setup(), and kvm_arch_check_processor_compat(), which
> all originated in x86 code from way back when, and needlessly complicate
> both common KVM code and architecture code. E.g. many architectures don't
> mark functions/data as __init/__ro_after_init purely because kvm_init()
> isn't marked __init to support x86's separate vendor modules.
>
> The idea/hope is that with those hooks gone (moved to arch code), it will
> be easier for x86 (and other architectures) to modify their module init
> sequences as needed without having to fight common KVM code. E.g. I'm
> hoping that ARM can build on this to simplify its hardware enabling logic,
> especially the pKVM side of things.
>
> There are bug fixes throughout this series. They are more scattered than
> I would usually prefer, but getting the sequencing correct was a gigantic
> pain for many of the x86 fixes due to needing to fix common code in order
> for the x86 fix to have any meaning. And while the bugs are often fatal,
> they aren't all that interesting for most users as they either require a
> malicious admin or broken hardware, i.e. aren't likely to be encountered
> by the vast majority of KVM users. So unless someone _really_ wants a
> particular fix isolated for backporting, I'm not planning on shuffling
> patches.
>
> Tested on x86. Lightly tested on arm64. Compile tested only on all other
> architectures.
Thanks for the patch series. I the rebased TDX KVM patch series and it worked.
Since cpu offline needs to be rejected in some cases(To keep at least one cpu
on a package), arch hook for cpu offline is needed.
I can keep it in TDX KVM patch series.
diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 23c0f4bc63f1..ef7bcb845d42 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -17,6 +17,7 @@ BUILD_BUG_ON(1)
KVM_X86_OP(hardware_enable)
KVM_X86_OP(hardware_disable)
KVM_X86_OP(hardware_unsetup)
+KVM_X86_OP_OPTIONAL_RET0(offline_cpu)
KVM_X86_OP(has_emulated_msr)
KVM_X86_OP(vcpu_after_set_cpuid)
KVM_X86_OP(is_vm_type_supported)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 496c7c6eaff9..c420409aa96f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1468,6 +1468,7 @@ struct kvm_x86_ops {
int (*hardware_enable)(void);
void (*hardware_disable)(void);
void (*hardware_unsetup)(void);
+ int (*offline_cpu)(void);
bool (*has_emulated_msr)(struct kvm *kvm, u32 index);
void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 2ed5a017f7bc..17c5d6a76c93 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -12039,6 +12039,11 @@ void kvm_arch_hardware_disable(void)
drop_user_return_notifiers();
}
+int kvm_arch_offline_cpu(unsigned int cpu)
+{
+ return static_call(kvm_x86_offline_cpu)();
+}
+
bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu)
{
return vcpu->kvm->arch.bsp_vcpu_id == vcpu->vcpu_id;
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 620489b9aa93..4df79443fd11 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1460,6 +1460,7 @@ static inline void kvm_create_vcpu_debugfs(struct kvm_vcpu *vcpu) {}
int kvm_arch_hardware_enable(void);
void kvm_arch_hardware_disable(void);
#endif
+int kvm_arch_offline_cpu(unsigned int cpu);
int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);
bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu);
int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index f6b6dcedaa0a..f770fdc662d0 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -5396,16 +5396,24 @@ static void hardware_disable_nolock(void *junk)
__this_cpu_write(hardware_enabled, false);
}
+__weak int kvm_arch_offline_cpu(unsigned int cpu)
+{
+ return 0;
+}
+
static int kvm_offline_cpu(unsigned int cpu)
{
+ int r = 0;
+
mutex_lock(&kvm_lock);
- if (kvm_usage_count) {
+ r = kvm_arch_offline_cpu(cpu);
+ if (!r && kvm_usage_count) {
preempt_disable();
hardware_disable_nolock(NULL);
preempt_enable();
}
mutex_unlock(&kvm_lock);
- return 0;
+ return r;
}
static void hardware_disable_all_nolock(void)
--
Isaku Yamahata <isaku.yamahata@gmail.com>
next prev parent reply other threads:[~2022-11-04 7:17 UTC|newest]
Thread overview: 127+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-02 23:18 [PATCH 00/44] KVM: Rework kvm_init() and hardware enabling Sean Christopherson
2022-11-02 23:18 ` [PATCH 01/44] KVM: Register /dev/kvm as the _very_ last thing during initialization Sean Christopherson
2022-11-02 23:18 ` [PATCH 02/44] KVM: Initialize IRQ FD after arch hardware setup Sean Christopherson
2022-11-04 0:41 ` Chao Gao
2022-11-04 20:15 ` Sean Christopherson
2022-11-02 23:18 ` [PATCH 03/44] KVM: Allocate cpus_hardware_enabled " Sean Christopherson
2022-11-04 5:37 ` Yuan Yao
2022-11-02 23:18 ` [PATCH 04/44] KVM: Teardown VFIO ops earlier in kvm_exit() Sean Christopherson
2022-11-03 12:46 ` Cornelia Huck
2022-11-07 17:56 ` Eric Farman
2022-11-02 23:18 ` [PATCH 05/44] KVM: s390: Unwind kvm_arch_init() piece-by-piece() if a step fails Sean Christopherson
2022-11-07 17:57 ` Eric Farman
2022-11-02 23:18 ` [PATCH 06/44] KVM: s390: Move hardware setup/unsetup to init/exit Sean Christopherson
2022-11-07 17:58 ` Eric Farman
2022-11-02 23:18 ` [PATCH 07/44] KVM: x86: Do timer initialization after XCR0 configuration Sean Christopherson
2022-11-02 23:18 ` [PATCH 08/44] KVM: x86: Move hardware setup/unsetup to init/exit Sean Christopherson
2022-11-04 6:22 ` Yuan Yao
2022-11-04 16:31 ` Sean Christopherson
2022-11-02 23:18 ` [PATCH 09/44] KVM: Drop arch hardware (un)setup hooks Sean Christopherson
2022-11-07 3:01 ` Anup Patel
2022-11-07 18:22 ` Eric Farman
2022-11-02 23:18 ` [PATCH 10/44] KVM: VMX: Clean up eVMCS enabling if KVM initialization fails Sean Christopherson
2022-11-03 14:01 ` Paolo Bonzini
2022-11-03 14:04 ` Paolo Bonzini
2022-11-03 14:28 ` Vitaly Kuznetsov
2022-11-11 1:38 ` Sean Christopherson
2022-11-15 9:30 ` Vitaly Kuznetsov
2022-11-02 23:18 ` [PATCH 11/44] KVM: x86: Move guts of kvm_arch_init() to standalone helper Sean Christopherson
2022-11-02 23:18 ` [PATCH 12/44] KVM: VMX: Do _all_ initialization before exposing /dev/kvm to userspace Sean Christopherson
2022-11-02 23:18 ` [PATCH 13/44] KVM: x86: Serialize vendor module initialization (hardware setup) Sean Christopherson
2022-11-16 1:46 ` Huang, Kai
2022-11-16 15:52 ` Sean Christopherson
2022-11-02 23:18 ` [PATCH 14/44] KVM: arm64: Simplify the CPUHP logic Sean Christopherson
2022-11-02 23:18 ` [PATCH 15/44] KVM: arm64: Free hypervisor allocations if vector slot init fails Sean Christopherson
2022-11-02 23:18 ` [PATCH 16/44] KVM: arm64: Unregister perf callbacks if hypervisor finalization fails Sean Christopherson
2022-11-02 23:18 ` [PATCH 17/44] KVM: arm64: Do arm/arch initialiation without bouncing through kvm_init() Sean Christopherson
2022-11-03 7:25 ` Philippe Mathieu-Daudé
2022-11-03 15:29 ` Sean Christopherson
2022-11-02 23:18 ` [PATCH 18/44] KVM: arm64: Mark kvm_arm_init() and its unique descendants as __init Sean Christopherson
2022-11-02 23:18 ` [PATCH 19/44] KVM: MIPS: Hardcode callbacks to hardware virtualization extensions Sean Christopherson
2022-11-02 23:18 ` [PATCH 20/44] KVM: MIPS: Setup VZ emulation? directly from kvm_mips_init() Sean Christopherson
2022-11-03 7:10 ` Philippe Mathieu-Daudé
2022-11-02 23:18 ` [PATCH 21/44] KVM: MIPS: Register die notifier prior to kvm_init() Sean Christopherson
2022-11-03 7:12 ` Philippe Mathieu-Daudé
2022-11-02 23:18 ` [PATCH 22/44] KVM: RISC-V: Do arch init directly in riscv_kvm_init() Sean Christopherson
2022-11-03 7:14 ` Philippe Mathieu-Daudé
2022-11-07 3:05 ` Anup Patel
2022-11-02 23:18 ` [PATCH 23/44] KVM: RISC-V: Tag init functions and data with __init, __ro_after_init Sean Christopherson
2022-11-07 3:10 ` Anup Patel
2022-11-02 23:18 ` [PATCH 24/44] KVM: PPC: Move processor compatibility check to module init Sean Christopherson
2022-11-02 23:18 ` [PATCH 25/44] KVM: s390: Do s390 specific init without bouncing through kvm_init() Sean Christopherson
2022-11-03 7:16 ` Philippe Mathieu-Daudé
2022-11-03 12:44 ` Claudio Imbrenda
2022-11-03 13:21 ` Claudio Imbrenda
2022-11-07 18:22 ` Eric Farman
2022-11-02 23:18 ` [PATCH 26/44] KVM: s390: Mark __kvm_s390_init() and its descendants as __init Sean Christopherson
2022-11-07 18:22 ` Eric Farman
2022-11-02 23:18 ` [PATCH 27/44] KVM: Drop kvm_arch_{init,exit}() hooks Sean Christopherson
2022-11-03 7:18 ` Philippe Mathieu-Daudé
2022-11-07 3:13 ` Anup Patel
2022-11-07 19:08 ` Eric Farman
2022-11-02 23:18 ` [PATCH 28/44] KVM: VMX: Make VMCS configuration/capabilities structs read-only after init Sean Christopherson
2022-11-02 23:18 ` [PATCH 29/44] KVM: x86: Do CPU compatibility checks in x86 code Sean Christopherson
2022-11-02 23:18 ` [PATCH 30/44] KVM: Drop kvm_arch_check_processor_compat() hook Sean Christopherson
2022-11-03 7:20 ` Philippe Mathieu-Daudé
2022-11-07 3:16 ` Anup Patel
2022-11-07 19:08 ` Eric Farman
2022-11-02 23:18 ` [PATCH 31/44] KVM: x86: Use KBUILD_MODNAME to specify vendor module name Sean Christopherson
2022-11-02 23:18 ` [PATCH 32/44] KVM: x86: Unify pr_fmt to use module name for all KVM modules Sean Christopherson
2022-11-10 7:31 ` Robert Hoo
2022-11-10 16:50 ` Sean Christopherson
2022-11-30 23:02 ` Sean Christopherson
2022-12-01 1:34 ` Robert Hoo
2022-11-02 23:19 ` [PATCH 33/44] KVM: x86: Do VMX/SVM support checks directly in vendor code Sean Christopherson
2022-11-03 15:08 ` Paolo Bonzini
2022-11-03 18:35 ` Sean Christopherson
2022-11-03 18:46 ` Paolo Bonzini
2022-11-03 18:58 ` Sean Christopherson
2022-11-04 8:02 ` Paolo Bonzini
2022-11-04 15:40 ` Sean Christopherson
2022-11-15 22:50 ` Huang, Kai
2022-11-16 1:56 ` Sean Christopherson
2022-11-02 23:19 ` [PATCH 34/44] KVM: VMX: Shuffle support checks and hardware enabling code around Sean Christopherson
2022-11-02 23:19 ` [PATCH 35/44] KVM: SVM: Check for SVM support in CPU compatibility checks Sean Christopherson
2022-11-02 23:19 ` [PATCH 36/44] KVM: x86: Do compatibility checks when onlining CPU Sean Christopherson
2022-11-03 15:17 ` Paolo Bonzini
2022-11-03 17:44 ` Sean Christopherson
2022-11-03 17:57 ` Paolo Bonzini
2022-11-03 21:04 ` Isaku Yamahata
2022-11-03 22:34 ` Sean Christopherson
2022-11-04 7:18 ` Isaku Yamahata
2022-11-11 0:06 ` Sean Christopherson
2022-11-02 23:19 ` [PATCH 37/44] KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section Sean Christopherson
2022-11-10 7:26 ` Robert Hoo
2022-11-10 16:49 ` Sean Christopherson
2022-11-02 23:19 ` [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Sean Christopherson
2022-11-10 1:08 ` Huang, Kai
2022-11-10 2:20 ` Huang, Kai
2022-11-10 1:33 ` Huang, Kai
2022-11-10 2:11 ` Huang, Kai
2022-11-10 16:58 ` Sean Christopherson
2022-11-15 20:16 ` Sean Christopherson
2022-11-15 20:21 ` Sean Christopherson
2022-11-16 12:23 ` Huang, Kai
2022-11-16 17:11 ` Sean Christopherson
2022-11-17 1:39 ` Huang, Kai
2022-11-17 15:16 ` Sean Christopherson
2022-11-02 23:19 ` [PATCH 39/44] KVM: Drop kvm_count_lock and instead protect kvm_usage_count with kvm_lock Sean Christopherson
2022-11-03 15:23 ` Paolo Bonzini
2022-11-03 17:53 ` Sean Christopherson
2022-11-02 23:19 ` [PATCH 40/44] KVM: Remove on_each_cpu(hardware_disable_nolock) in kvm_exit() Sean Christopherson
2022-11-02 23:19 ` [PATCH 41/44] KVM: Use a per-CPU variable to track which CPUs have enabled virtualization Sean Christopherson
2022-11-02 23:19 ` [PATCH 42/44] KVM: Make hardware_enable_failed a local variable in the "enable all" path Sean Christopherson
2022-11-02 23:19 ` [PATCH 43/44] KVM: Register syscore (suspend/resume) ops early in kvm_init() Sean Christopherson
2022-11-02 23:19 ` [PATCH 44/44] KVM: Opt out of generic hardware enabling on s390 and PPC Sean Christopherson
2022-11-07 3:23 ` Anup Patel
2022-11-03 12:08 ` [PATCH 00/44] KVM: Rework kvm_init() and hardware enabling Christian Borntraeger
2022-11-03 15:27 ` Paolo Bonzini
2022-11-04 7:17 ` Isaku Yamahata [this message]
2022-11-04 7:59 ` Paolo Bonzini
2022-11-04 20:27 ` Sean Christopherson
2022-11-07 21:46 ` Isaku Yamahata
2022-11-08 1:09 ` Huang, Kai
2022-11-08 5:43 ` Isaku Yamahata
2022-11-08 8:56 ` Huang, Kai
2022-11-08 10:35 ` Huang, Kai
2022-11-08 17:46 ` Sean Christopherson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221104071749.GC1063309@ls.amr.corp.intel.com \
--to=isaku.yamahata@gmail.com \
--cc=aleksandar.qemu.devel@gmail.com \
--cc=alexandru.elisei@arm.com \
--cc=anup@brainfault.org \
--cc=aou@eecs.berkeley.edu \
--cc=atishp@atishpatra.org \
--cc=borntraeger@linux.ibm.com \
--cc=chao.gao@intel.com \
--cc=chenhuacai@kernel.org \
--cc=david@redhat.com \
--cc=farman@linux.ibm.com \
--cc=farosas@linux.ibm.com \
--cc=frankja@linux.ibm.com \
--cc=imbrenda@linux.ibm.com \
--cc=isaku.yamahata@intel.com \
--cc=james.morse@arm.com \
--cc=kvm-riscv@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=kvmarm@lists.linux.dev \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=linux-s390@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=maz@kernel.org \
--cc=mjrosato@linux.ibm.com \
--cc=mpe@ellerman.id.au \
--cc=oliver.upton@linux.dev \
--cc=palmer@dabbelt.com \
--cc=paul.walmsley@sifive.com \
--cc=pbonzini@redhat.com \
--cc=seanjc@google.com \
--cc=suzuki.poulose@arm.com \
--cc=tglx@linutronix.de \
--cc=vkuznets@redhat.com \
--cc=yuan.yao@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).