linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Isaku Yamahata <isaku.yamahata@gmail.com>
To: Sagi Shahar <sagis@google.com>
Cc: isaku.yamahata@intel.com, kvm@vger.kernel.org,
	linux-kernel@vger.kernel.org, isaku.yamahata@gmail.com,
	Paolo Bonzini <pbonzini@redhat.com>,
	erdemaktas@google.com, Sean Christopherson <seanjc@google.com>,
	Sean Christopherson <sean.j.christopherson@intel.com>,
	Kai Huang <kai.huang@intel.com>
Subject: Re: [PATCH v9 016/105] KVM: TDX: create/destroy VM structure
Date: Thu, 13 Oct 2022 01:55:54 -0700	[thread overview]
Message-ID: <20221013085554.GA2756200@ls.amr.corp.intel.com> (raw)
In-Reply-To: <CAAhR5DFrwP+5K8MOxz5YK7jYShhaK4A+2h1Pi31U_9+Z+cz-0A@mail.gmail.com>

On Wed, Oct 12, 2022 at 03:30:26PM -0700,
Sagi Shahar <sagis@google.com> wrote:


> > +int tdx_vm_init(struct kvm *kvm)
> > +{
> > +       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
> > +       cpumask_var_t packages;
> > +       int ret, i;
> > +       u64 err;
> > +
> > +       /* vCPUs can't be created until after KVM_TDX_INIT_VM. */
> > +       kvm->max_vcpus = 0;
> 
> The fact that vCPUs can't be created until KVM_TDX_INIT_VM is called
> will make it difficult to implement intra host migration. See longer
> discussion below.
...
> Me, Sean and Isaku had a short discussion offline regarding the
> interaction between the proposed API in this patch and intra-host
> migration. To summarize:
> 
> For intra-host migration you generally want the destination VM to be
> initialized including the right number of vCPUs before you migrate the
> source VM state into it.
> The proposed API makes it difficult since it forces the destination VM
> to call KVM_TDX_INIT_VM before creating vCPUs which initializes TDX
> state and allocate a new hkid for the destination VM which would never
> be used. This can create a resource limitation on migrating VMs where
> there shouldn't be one.
> 
> To solve this issue there are 2 main proposed changes to the API:
> 
> 1. Add a new API based on ioctl(KVM_ENABLE_CAP) to let userspace
> modify the max number of vcpus:
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 43a6a7efc6ec..6055098b025b 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -6278,6 +6278,18 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
>                 }
>                 mutex_unlock(&kvm->lock);
>                 break;
> +       case KVM_CAP_MAX_VCPUS:
> +               r = -EINVAL;
> +               if (cap->args[0] > KVM_MAX_VCPUS)
> +                       break;
> +
> +               mutex_lock(&kvm->lock);
> +               if (!kvm->created_vcpus) {
> +                       kvm->max_vcpus = cap->args[0];
> +                       r = 0;
> +               }
> +               mutex_unlock(&kvm->lock);
> +               break;
>         case KVM_CAP_MAX_VCPU_ID:
>                 r = -EINVAL;
>                 if (cap->args[0] > KVM_MAX_VCPU_IDS)
> 
> 2. Modify the existing API such that max_vcpus will be set to
> KVM_MAX_VCPUS like in regular VMs and during KVM_TDX_INIT_VM, if the
> user created more vCPUs than the number specified, KVM_TDX_INIT_VM
> will fail.
> 
> For option (1), there are some possible variations:
> 1.a. Do we keep the max_vcpus argument in KVM_TDX_INIT_VM? If so, we
> need to check if max_vcpus matches the number of max_vcpus already set
> and fail otherwise.
> 1.b. Do we require KVM_ENABLE_CAP_VM(KVM_CAP_MAX_VCPUS) to be called?
> Theoretically, we can set max_vcpus to the KVM default KVM_MAX_VCPUS
> and allow the user to change it as long as vcpus hasn't been created.
> If KVM_ENABLE_CAP_VM(KVM_CAP_MAX_VCPUS), the behavior will remain the
> same as regular VMs right now.
> 
> In my opinion, the cleanest solution would be option 1 (new
> KVM_CAP_MAX_VCPUS API) while removing the max_vcpus argument from
> KVM_TDX_INIT_VM and setting the initial max_vcpus to KVM_MAX_VCPUS and
> not requiring the new ioctl to be called unless userspace wants to
> specifically limit the number of vcpus. In that case,
> KVM_CAP_MAX_VCPUS can be called at any time until vcpus are created.

Regarding to KVM_CAP_MAX_CPUS vs KVM_TDX_INIT_VM, KVM_CAP_MAX_CPUS is more
generic, KVM_CAP_MAX_CPUS  would be better.  This follows tsc frequency.

If option (1) is adapted, the logic should go to the common code, i.e. under
linux/virt/kvm/, because there is nothing specific to x86.  I don't see any use
case other than TDX, though.
-- 
Isaku Yamahata <isaku.yamahata@gmail.com>

  reply	other threads:[~2022-10-13  8:56 UTC|newest]

Thread overview: 111+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-30 10:16 [PATCH v9 000/105] KVM TDX basic feature support isaku.yamahata
2022-09-30 10:16 ` [PATCH v9 001/105] KVM: VMX: Move out vmx_x86_ops to 'main.c' to wrap VMX and TDX isaku.yamahata
2022-09-30 10:16 ` [PATCH v9 002/105] KVM: x86: Refactor KVM VMX module init/exit functions isaku.yamahata
2022-09-30 10:16 ` [PATCH v9 003/105] KVM: TDX: Add placeholders for TDX VM/vcpu structure isaku.yamahata
2022-09-30 10:16 ` [PATCH v9 004/105] x86/virt/tdx: Add a helper function to return system wide info about TDX module isaku.yamahata
2022-09-30 10:16 ` [PATCH v9 005/105] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 006/105] KVM: x86: Introduce vm_type to differentiate default VMs from confidential VMs isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 007/105] KVM: TDX: Make TDX VM type supported isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 008/105] [MARKER] The start of TDX KVM patch series: TDX architectural definitions isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 009/105] KVM: TDX: Define " isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 010/105] KVM: TDX: Add TDX "architectural" error codes isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 011/105] KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 012/105] KVM: TDX: Add helper functions to print TDX SEAMCALL error isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 013/105] [MARKER] The start of TDX KVM patch series: TD VM creation/destruction isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 014/105] KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 015/105] x86/cpu: Add helper functions to allocate/free TDX private host key id isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 016/105] KVM: TDX: create/destroy VM structure isaku.yamahata
2022-10-12 22:30   ` Sagi Shahar
2022-10-13  8:55     ` Isaku Yamahata [this message]
2022-09-30 10:17 ` [PATCH v9 017/105] KVM: TDX: Refuse to unplug the last cpu on the package isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 018/105] KVM: TDX: x86: Add ioctl to get TDX systemwide parameters isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 019/105] KVM: TDX: Add place holder for TDX VM specific mem_enc_op ioctl isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 020/105] KVM: TDX: initialize VM with TDX specific parameters isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 021/105] KVM: TDX: Make pmu_intel.c ignore guest TD case isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 022/105] [MARKER] The start of TDX KVM patch series: TD vcpu creation/destruction isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 023/105] KVM: TDX: allocate/free TDX vcpu structure isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 024/105] KVM: TDX: Do TDX specific vcpu initialization isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 025/105] KVM: TDX: Use private memory for TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 026/105] [MARKER] The start of TDX KVM patch series: KVM MMU GPA shared bits isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 027/105] KVM: x86/mmu: introduce config for PRIVATE KVM MMU isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 028/105] KVM: x86/mmu: Add address conversion functions for TDX shared bit of GPA isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 029/105] [MARKER] The start of TDX KVM patch series: KVM TDP refactoring for TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 030/105] KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 031/105] KVM: x86/mmu: Make sync_page not use hard-coded 0 as the initial SPTE value isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 032/105] KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed SPTE isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 033/105] KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_{value, mask} isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 034/105] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 035/105] KVM: TDX: Enable mmio spte caching always for TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 036/105] KVM: x86/mmu: Disallow fast page fault on private GPA isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 037/105] KVM: x86/mmu: Allow per-VM override of the TDP max page level isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 038/105] KVM: VMX: Introduce test mode related to EPT violation VE isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 039/105] [MARKER] The start of TDX KVM patch series: KVM TDP MMU hooks isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 040/105] KVM: x86/tdp_mmu: refactor kvm_tdp_mmu_map() isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 041/105] KVM: x86/tdp_mmu: Init role member of struct kvm_mmu_page at allocation isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 042/105] KVM: x86/mmu: Require TDP MMU for TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 043/105] KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 044/105] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 045/105] KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 046/105] KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 047/105] [MARKER] The start of TDX KVM patch series: TDX EPT violation isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 048/105] KVM: x86/mmu: Disallow dirty logging for x86 TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 049/105] KVM: x86/tdp_mmu: Ignore unsupported mmu operation on private GFNs isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 050/105] KVM: VMX: Split out guts of EPT violation to common/exposed function isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 051/105] KVM: VMX: Move setting of EPT MMU masks to common VT-x code isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 052/105] KVM: TDX: Add load_mmu_pgd method for TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 053/105] KVM: TDX: don't request KVM_REQ_APIC_PAGE_RELOAD isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 054/105] KVM: x86/VMX: introduce vmx tlb_remote_flush and tlb_remote_flush_with_range isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 055/105] KVM: TDX: TDP MMU TDX support isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 056/105] [MARKER] The start of TDX KVM patch series: KVM TDP MMU MapGPA isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 057/105] KVM: Add functions to set GFN to private or shared isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 058/105] KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 059/105] KVM: x86/tdp_mmu: implement MapGPA hypercall for TDX isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 060/105] [MARKER] The start of TDX KVM patch series: TD finalization isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 061/105] KVM: TDX: Create initial guest memory isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 062/105] KVM: TDX: Finalize VM initialization isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 063/105] [MARKER] The start of TDX KVM patch series: TD vcpu enter/exit isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 064/105] KVM: TDX: Add helper assembly function to TDX vcpu isaku.yamahata
2022-09-30 10:17 ` [PATCH v9 065/105] KVM: TDX: Implement TDX vcpu enter/exit path isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 066/105] KVM: TDX: vcpu_run: save/restore host state(host kernel gs) isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 067/105] KVM: TDX: restore host xsave state when exit from the guest TD isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 068/105] KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 069/105] KVM: TDX: restore user ret MSRs isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 070/105] [MARKER] The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 071/105] KVM: TDX: complete interrupts after tdexit isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 072/105] KVM: TDX: restore debug store when TD exit isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 073/105] KVM: TDX: handle vcpu migration over logical processor isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 074/105] KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 075/105] KVM: TDX: Add support for find pending IRQ in a protected local APIC isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 076/105] KVM: x86: Assume timer IRQ was injected if APIC state is proteced isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 077/105] KVM: TDX: remove use of struct vcpu_vmx from posted_interrupt.c isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 078/105] KVM: TDX: Implement interrupt injection isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 079/105] KVM: TDX: Implements vcpu request_immediate_exit isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 080/105] KVM: TDX: Implement methods to inject NMI isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 081/105] KVM: VMX: Modify NMI and INTR handlers to take intr_info as function argument isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 082/105] KVM: VMX: Move NMI/exception handler to common helper isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 083/105] KVM: x86: Split core of hypercall emulation to helper function isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 084/105] KVM: TDX: Add a place holder to handle TDX VM exit isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 085/105] KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 086/105] KVM: TDX: handle EXIT_REASON_OTHER_SMI isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 087/105] KVM: TDX: handle ept violation/misconfig exit isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 088/105] KVM: TDX: handle EXCEPTION_NMI and EXTERNAL_INTERRUPT isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 089/105] KVM: TDX: Add a place holder for handler of TDX hypercalls (TDG.VP.VMCALL) isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 090/105] KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 091/105] KVM: TDX: Handle TDX PV CPUID hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 092/105] KVM: TDX: Handle TDX PV HLT hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 093/105] KVM: TDX: Handle TDX PV port io hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 094/105] KVM: TDX: Handle TDX PV MMIO hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 095/105] KVM: TDX: Implement callbacks for MSR operations for TDX isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 096/105] KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 097/105] KVM: TDX: Handle TDX PV report fatal error hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 098/105] KVM: TDX: Handle TDX PV map_gpa hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 099/105] KVM: TDX: Handle TDG.VP.VMCALL<GetTdVmCallInfo> hypercall isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 100/105] KVM: TDX: Silently discard SMI request isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 101/105] KVM: TDX: Silently ignore INIT/SIPI isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 102/105] KVM: TDX: Add methods to ignore accesses to CPU state isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 103/105] Documentation/virt/kvm: Document on Trust Domain Extensions(TDX) isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 104/105] KVM: x86: design documentation on TDX support of x86 KVM TDP MMU isaku.yamahata
2022-09-30 10:18 ` [PATCH v9 105/105] [MARKER] the end of (the first phase of) TDX KVM patch series isaku.yamahata
2022-10-01  8:30 ` [PATCH v9 000/105] KVM TDX basic feature support Bagas Sanjaya
2022-10-03 18:29   ` Isaku Yamahata
2022-10-03 20:08     ` Huang, Kai

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20221013085554.GA2756200@ls.amr.corp.intel.com \
    --to=isaku.yamahata@gmail.com \
    --cc=erdemaktas@google.com \
    --cc=isaku.yamahata@intel.com \
    --cc=kai.huang@intel.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=sagis@google.com \
    --cc=sean.j.christopherson@intel.com \
    --cc=seanjc@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).