Bugs in kvm guest migration

Bugs in kvm guest migration

[Jan Kiszka]

Hi,

my migration issues currently look like they are driven by at least two
bugs (that's real fun). One slipped into qemu-kvm during merge of
upstream, the other is either an issue of current kvm.git master or
kvm-kmod's wrapping of the latter.

Problem 1
---------
Setup: qemu-kvm after 249cdd049c7976614ad46eb86409d5e1e53835cc (merge
f9ea97e859dc0fce26c35f7e047a51788a3e5441 is the first usable one again,
and it's affected), any kernel module version, 64-bit host & guest (Linux).
Effect: After resume of the migration target, its timer is still ticking
(APIC IRQs arrive) but timed Linux tasks make no progress. Linux guest
uses highres/nohz, when disabling both, the problem disappears.

Problem 2
---------
Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host & guest.
Effect: The migration target either locks up or reboots immediately.
I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
kvm module build before that works, everything including and after
26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
between crashes the host for known reasons). It's still unclear if this
is an kvm-kmod wrapping issue of the user-space return notifiers. Will
check once problem #1 is understood.

Any early ideas on this are welcome while I continue digging.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Bugs in kvm guest migration

[Jan Kiszka]

Hi,

my migration issues currently look like they are driven by at least two
bugs (that's real fun). One slipped into qemu-kvm during merge of
upstream, the other is either an issue of current kvm.git master or
kvm-kmod's wrapping of the latter.

Problem 1
---------
Setup: qemu-kvm after 249cdd049c7976614ad46eb86409d5e1e53835cc (merge
f9ea97e859dc0fce26c35f7e047a51788a3e5441 is the first usable one again,
and it's affected), any kernel module version, 64-bit host & guest (Linux).
Effect: After resume of the migration target, its timer is still ticking
(APIC IRQs arrive) but timed Linux tasks make no progress. Linux guest
uses highres/nohz, when disabling both, the problem disappears.

Problem 2
---------
Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host & guest.
Effect: The migration target either locks up or reboots immediately.
I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
kvm module build before that works, everything including and after
26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
between crashes the host for known reasons). It's still unclear if this
is an kvm-kmod wrapping issue of the user-space return notifiers. Will
check once problem #1 is understood.

Any early ideas on this are welcome while I continue digging.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Bugs in kvm guest migration

[Jan Kiszka]

Jan Kiszka wrote:
> Problem 2
> ---------
> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host & guest.
> Effect: The migration target either locks up or reboots immediately.
> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
> kvm module build before that works, everything including and after
> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
> between crashes the host for known reasons). It's still unclear if this
> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
> check once problem #1 is understood.

It is kernel-related, using kvm.git as host kernel makes no difference.

This may now mean that kvm is buggy or that it triggers some msr
save/restore related issues in qemu. Digging even deeper...

Jan

[Qemu-devel] Re: Bugs in kvm guest migration

[Jan Kiszka]

Jan Kiszka wrote:
> Problem 2
> ---------
> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host & guest.
> Effect: The migration target either locks up or reboots immediately.
> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
> kvm module build before that works, everything including and after
> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
> between crashes the host for known reasons). It's still unclear if this
> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
> check once problem #1 is understood.

It is kernel-related, using kvm.git as host kernel makes no difference.

This may now mean that kvm is buggy or that it triggers some msr
save/restore related issues in qemu. Digging even deeper...

Jan

Re: Bugs in kvm guest migration

[Avi Kivity]

On 12/03/2009 09:23 PM, Jan Kiszka wrote:
> Jan Kiszka wrote:
>    
>> Problem 2
>> ---------
>> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host&  guest.
>> Effect: The migration target either locks up or reboots immediately.
>> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
>> kvm module build before that works, everything including and after
>> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
>> between crashes the host for known reasons). It's still unclear if this
>> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
>> check once problem #1 is understood.
>>      
> It is kernel-related, using kvm.git as host kernel makes no difference.
>
> This may now mean that kvm is buggy or that it triggers some msr
> save/restore related issues in qemu. Digging even deeper...
>    

Does calling drop_user_return_notifiers() (static in x86.c, will need 
exporting) in vmx_load_host_state() within the preempt-disable region help?

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

[Qemu-devel] Re: Bugs in kvm guest migration

[Avi Kivity]

On 12/03/2009 09:23 PM, Jan Kiszka wrote:
> Jan Kiszka wrote:
>    
>> Problem 2
>> ---------
>> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host&  guest.
>> Effect: The migration target either locks up or reboots immediately.
>> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
>> kvm module build before that works, everything including and after
>> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
>> between crashes the host for known reasons). It's still unclear if this
>> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
>> check once problem #1 is understood.
>>      
> It is kernel-related, using kvm.git as host kernel makes no difference.
>
> This may now mean that kvm is buggy or that it triggers some msr
> save/restore related issues in qemu. Digging even deeper...
>    

Does calling drop_user_return_notifiers() (static in x86.c, will need 
exporting) in vmx_load_host_state() within the preempt-disable region help?

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: Bugs in kvm guest migration

[Jan Kiszka]

Avi Kivity wrote:
> On 12/03/2009 09:23 PM, Jan Kiszka wrote:
>> Jan Kiszka wrote:
>>    
>>> Problem 2
>>> ---------
>>> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host&  guest.
>>> Effect: The migration target either locks up or reboots immediately.
>>> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
>>> kvm module build before that works, everything including and after
>>> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
>>> between crashes the host for known reasons). It's still unclear if this
>>> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
>>> check once problem #1 is understood.
>>>      
>> It is kernel-related, using kvm.git as host kernel makes no difference.
>>
>> This may now mean that kvm is buggy or that it triggers some msr
>> save/restore related issues in qemu. Digging even deeper...
>>    
> 
> Does calling drop_user_return_notifiers() (static in x86.c, will need 
> exporting) in vmx_load_host_state() within the preempt-disable region help?
> 

Nope.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: Bugs in kvm guest migration

[Jan Kiszka]

Avi Kivity wrote:
> On 12/03/2009 09:23 PM, Jan Kiszka wrote:
>> Jan Kiszka wrote:
>>    
>>> Problem 2
>>> ---------
>>> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host&  guest.
>>> Effect: The migration target either locks up or reboots immediately.
>>> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
>>> kvm module build before that works, everything including and after
>>> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
>>> between crashes the host for known reasons). It's still unclear if this
>>> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
>>> check once problem #1 is understood.
>>>      
>> It is kernel-related, using kvm.git as host kernel makes no difference.
>>
>> This may now mean that kvm is buggy or that it triggers some msr
>> save/restore related issues in qemu. Digging even deeper...
>>    
> 
> Does calling drop_user_return_notifiers() (static in x86.c, will need 
> exporting) in vmx_load_host_state() within the preempt-disable region help?
> 

Nope.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Bugs in kvm guest migration

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

Jan Kiszka wrote:
> Avi Kivity wrote:
>> On 12/03/2009 09:23 PM, Jan Kiszka wrote:
>>> Jan Kiszka wrote:
>>>    
>>>> Problem 2
>>>> ---------
>>>> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host&  guest.
>>>> Effect: The migration target either locks up or reboots immediately.
>>>> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
>>>> kvm module build before that works, everything including and after
>>>> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
>>>> between crashes the host for known reasons). It's still unclear if this
>>>> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
>>>> check once problem #1 is understood.
>>>>      
>>> It is kernel-related, using kvm.git as host kernel makes no difference.
>>>
>>> This may now mean that kvm is buggy or that it triggers some msr
>>> save/restore related issues in qemu. Digging even deeper...
>>>    
>> Does calling drop_user_return_notifiers() (static in x86.c, will need 
>> exporting) in vmx_load_host_state() within the preempt-disable region help?
>>
> 
> Nope.
> 

It was a qemu bug, resolved by "KVM: x86: Fix initial kvm_has_msr_star".

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] Re: Bugs in kvm guest migration

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

Jan Kiszka wrote:
> Avi Kivity wrote:
>> On 12/03/2009 09:23 PM, Jan Kiszka wrote:
>>> Jan Kiszka wrote:
>>>    
>>>> Problem 2
>>>> ---------
>>>> Setup: qemu head with vmstate fixes. kvm-kmod master, 64-bit host&  guest.
>>>> Effect: The migration target either locks up or reboots immediately.
>>>> I've nailed this down to 84d0b66c778d881eafca2a5d0d66678211c4e861. Every
>>>> kvm module build before that works, everything including and after
>>>> 26ede77f536d1bb369527a96c7fe7fdc8ba2f890 shows the effect (everything in
>>>> between crashes the host for known reasons). It's still unclear if this
>>>> is an kvm-kmod wrapping issue of the user-space return notifiers. Will
>>>> check once problem #1 is understood.
>>>>      
>>> It is kernel-related, using kvm.git as host kernel makes no difference.
>>>
>>> This may now mean that kvm is buggy or that it triggers some msr
>>> save/restore related issues in qemu. Digging even deeper...
>>>    
>> Does calling drop_user_return_notifiers() (static in x86.c, will need 
>> exporting) in vmx_load_host_state() within the preempt-disable region help?
>>
> 
> Nope.
> 

It was a qemu bug, resolved by "KVM: x86: Fix initial kvm_has_msr_star".

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]