Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Vitaly Kuznetsov <vkuznets@redhat.com>]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 23.07.20 14:52, Dr. David Alan Gilbert wrote:
>> * Vitaly Kuznetsov (vkuznets@redhat.com) wrote:
>>> Philippe Mathieu-Daudé <philmd@redhat.com> writes:
>>>
>>>> +Vitaly
>>>>
>>>> On 7/23/20 10:40 AM, Dr. David Alan Gilbert wrote:
>>>>> * Eduardo Habkost (ehabkost@redhat.com) wrote:
>>>>>> On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
>>>>>>> On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
>>>>>>>> On 22.07.20 19:35, Eduardo Habkost wrote:
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> What was the last version where it worked for you?  Does using
>>>>>>>>> "-cpu host,-vmx" help?
>>>>>>>>
>>>>>>>> Yeah, -vmx does indeed help.
>>>>>>>>
>>>>>>>> I didn't have the time to bisect yet. Just check my reflog, picked
>>>>>>>> eb6490f544, and that works.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> I could reproduce it locally[1], I will bisect it.
>>>>>>>
>>>>>>> The good news is that "-cpu host,+vmx" still works, on commit
>>>>>>> eb6490f544.
>>>>>>>
>>>>>>> [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
>>>>>>
>>>>>> Bisected to:
>>>>>>
>>>>>> commit b16c0e20c74218f2d69710cedad11da7dd4d2190
>>>>>> Author: Paolo Bonzini <pbonzini@redhat.com>
>>>>>> Date:   Wed May 20 10:49:22 2020 -0400
>>>>>>
>>>>>>      KVM: add support for AMD nested live migration
>>>>>>
>>>>>>      Support for nested guest live migration is part of Linux 5.8, add the
>>>>>>      corresponding code to QEMU.  The migration format consists of a few
>>>>>>      flags, is an opaque 4k blob.
>>>>>>
>>>>>>      The blob is in VMCB format (the control area represents the L1 VMCB
>>>>>>      control fields, the save area represents the pre-vmentry state; KVM does
>>>>>>      not use the host save area since the AMD manual allows that) but QEMU
>>>>>>      does not really care about that.  However, the flags need to be
>>>>>>      copied to hflags/hflags2 and back.
>>>>>>
>>>>>>      In addition, support for retrieving and setting the AMD nested virtualization
>>>>>>      states allows the L1 guest to be reset while running a nested guest, but
>>>>>>      a small bug in CPU reset needs to be fixed for that to work.
>>>>>>
>>>>>>      Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>>>>>
>>>>> Guesswork led me to try reverting the chunk in kvm_put_nested_state;
>>>>> without it the reset seems to work; I can't explain that code though.
>>>>>
>>>
>>> (sorry, missed the beginning of the discussion)
>>>
>>> So one does:
>>>
>>> (qemu) system_reset
>>>
>>> on Intel wiht '-cpu host' and the result is:
>>>
>>> (qemu) KVM: entry failed, hardware error 0x80000021
>> 
>> Interesting; I hadn't seen that error - I just see a hard hung guest
>> rather than a reset one.
>
> I've seen it once or twice, maybe that was also with a more complex 
> command line. The point is that an invalid state is very likely loaded 
> on reset. Not all invalid states cause KVM to complain, though. Some 
> just lock up the guest.

This depends on whether the guest has performed VMXON or not I believe.

Anyways, I *think* the fix will be:

diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index 2b6b744..75c2e68 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -3883,7 +3883,7 @@ static int kvm_put_nested_state(X86CPU *cpu)
     } else {
         env->nested_state->flags &= ~KVM_STATE_NESTED_GUEST_MODE;
     }
-    if (env->hflags2 & HF2_GIF_MASK) {
+    if (cpu_has_svm(env) && (env->hflags2 & HF2_GIF_MASK)) {
         env->nested_state->flags |= KVM_STATE_NESTED_GIF_SET;
     } else {
         env->nested_state->flags &= ~KVM_STATE_NESTED_GIF_SET;

As "KVM_STATE_NESTED_GIF_SET" is not relevant to nVMX, this works for me
but let me explore kernel side of this a bit more.

-- 
Vitaly