Re: [QEMU PATCH v2 0/2]: KVM: i386: Add support for save and restore nested state

From: Liran Alon <liran.alon@oracle.com>
To: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: ehabkost@redhat.com, kvm@vger.kernel.org, mtosatti@redhat.com,
	qemu-devel@nongnu.org, Paolo Bonzini <pbonzini@redhat.com>,
	rth@twiddle.net, jmattson@google.com
Subject: Re: [QEMU PATCH v2 0/2]: KVM: i386: Add support for save and restore nested state
Date: Tue, 13 Nov 2018 01:58:46 +0200	[thread overview]
Message-ID: <B6AEEFF2-BAF1-439A-8E44-51352C9E05AE@oracle.com> (raw)
In-Reply-To: <20181112165053.GF2293@work-vm>

> On 12 Nov 2018, at 18:50, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
> 
> * Daniel P. Berrangé (berrange@redhat.com) wrote:
>> On Sun, Nov 04, 2018 at 11:19:57PM +0100, Paolo Bonzini wrote:
>>> On 02/11/2018 17:54, Daniel P. Berrangé wrote:
>>>> We have usually followed a rule that new machine types must not
>>>> affect runability of a VM on a host. IOW new machine types should
>>>> not introduce dependancies on specific kernels, or hardware features
>>>> such as CPU flags.
>>> 
>>>> Anything that requires a new kernel feature thus ought to be an
>>>> opt-in config tunable on the CLI, separate from machine type
>>>> choice.
>>> 
>>> Unless someone tinkered with the module parameters, they could not even
>>> use nested virtualization before 4.20.  So for everyone else, "-cpu
>>> ...,+vmx" does count as an "opt-in config tunable on the CLI" that
>>> requires 4.20.
>>> 
>>> For those that did tinker with module parameters, we can grandfather in
>>> the old machine types, so that they can use nested virtualization with
>>> no live migration support.  For those that did not, however, I don't
>>> think it makes sense to say "oh by the way I really want to be able to
>>> migrate this VM" on the command line, or even worse on the monitor.
>> 
>> IIUC, 4.20 is only required from POV of migration state. Is it thus
>> possible to just register a migration blocker if QEMU is launched
>> on a host with kernel < 4.20.
>> 
>> Migration has always been busted historically, so those people using
>> nested VMX already won't be hurt by not having ability to live migrate
>> their VM, but could otherwise continue using them without being forced
>> to upgrade their kernel to fix a feature they're not even using.
> 
> Yes, although I am a bit worried we might have a population of users
> that:
>   a) Have enabled nesting
>   b) Run VMs with vmx enabled
>   c) Don't normally actually run nested guests
>   d) Currently happily migrate.
> 
> Dave

First of all, we should put the entire kvm_nested_state in a VMState subsection that has a .needed() method
that checks if ((format==VMX) && (vmx.vmxon_pa != -1ull));
This will allow migrating a VM that has nested exposed but didn’t really enter VMX operation to still be able
to successfully migrate to old hosts without any issues.

The only problem remaining to be solved that you discuss here is what happens if user runs with modern QEMU
(Including my to-be-written v3 patches discussed here) on a host that has kernel without KVM_CAP_NESTED_STATE.
In this kernel, it is impossible for userspace (e.g. QEMU) to know if guest that was exposed with VMX actually used it.
So for those cases, I see no option but to modify QEMU to also say that if guest is exposed with VMX and
we are running on kernel without KVM_CAP_NESTED_STATE, then this is a migration blocker.

-Liran

> 
>> Regards,
>> Daniel
>> -- 
>> |: https://urldefense.proofpoint.com/v2/url?u=https-3A__berrange.com&d=DwIDAw&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=ximE6x4FrQ5vo3D4wF3w-cVsKgtNs85wCTL5GiP_B5Q&s=rs9ZkUUS37SHrs_oZJ9uIiXtpXvUkJBpfVEe9OSiQzk&e=      -o-    https://urldefense.proofpoint.com/v2/url?u=https-3A__www.flickr.com_photos_dberrange&d=DwIDAw&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=ximE6x4FrQ5vo3D4wF3w-cVsKgtNs85wCTL5GiP_B5Q&s=u_hnAFyY8BVwto0FsomTcZ3dmLKPYb1hwI_jRXI6EZg&e= :|
>> |: https://urldefense.proofpoint.com/v2/url?u=https-3A__libvirt.org&d=DwIDAw&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=ximE6x4FrQ5vo3D4wF3w-cVsKgtNs85wCTL5GiP_B5Q&s=SEIw983ixpJUUySxUwrxtIbnjvHTB9ff3MaqULaulQw&e=         -o-            https://urldefense.proofpoint.com/v2/url?u=https-3A__fstop138.berrange.com&d=DwIDAw&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=ximE6x4FrQ5vo3D4wF3w-cVsKgtNs85wCTL5GiP_B5Q&s=yvxjeOrwjXjf08RBhPdX53lJN1W-8WSXT25ZeMSA06k&e= :|
>> |: https://urldefense.proofpoint.com/v2/url?u=https-3A__entangle-2Dphoto.org&d=DwIDAw&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=ximE6x4FrQ5vo3D4wF3w-cVsKgtNs85wCTL5GiP_B5Q&s=9TIjtmf6AVFYWbyzI5vl-zXTaNCSCMAxyck92pc8yvY&e=    -o-    https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instagram.com_dberrange&d=DwIDAw&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=Jk6Q8nNzkQ6LJ6g42qARkg6ryIDGQr-yKXPNGZbpTx0&m=ximE6x4FrQ5vo3D4wF3w-cVsKgtNs85wCTL5GiP_B5Q&s=Wapdtm0yT4j-9EjMoxwo9QvRZ3h9Fk_CvpQq8J4TDpg&e= :|
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK