[Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Stefan Priebe - Profihost AG]

Hello,

today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
is running windows and virtio-balloon and with machine type pc-i440fx-2.5.

The output of the target qemu process was:
kvm_apic_post_load: Yeh
kvm_apic_post_load: Yeh
kvm_apic_post_load: Yeh
kvm_apic_post_load: Yeh
kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
kvm: error while loading state for instance 0x0 of device
'0000:00:03.0/virtio-balloon'
kvm: load of migration failed: Operation not permitted

Greets,
Stefan

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Dr. David Alan Gilbert]

* Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
> Hello,
> 
> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
> 
> The output of the target qemu process was:
> kvm_apic_post_load: Yeh
> kvm_apic_post_load: Yeh
> kvm_apic_post_load: Yeh
> kvm_apic_post_load: Yeh
> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
> kvm: error while loading state for instance 0x0 of device
> '0000:00:03.0/virtio-balloon'
> kvm: load of migration failed: Operation not permitted

Yes that's a known bug; only seems to affect windows guests, and I believe
doesn't even need to cross versions.

There's a bunch of fixes that Stefan applied to virtio code
that I think fix this; I see that he cc'd qemu-stable.
I think it's 4b7f91ed, but I'm not sure if there are others needed.

Dave
> 
> Greets,
> Stefan
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Stefan Priebe - Profihost AG]

Hello,

Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert:
> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>> Hello,
>>
>> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
>> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
>>
>> The output of the target qemu process was:
>> kvm_apic_post_load: Yeh
>> kvm_apic_post_load: Yeh
>> kvm_apic_post_load: Yeh
>> kvm_apic_post_load: Yeh
>> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
>> kvm: error while loading state for instance 0x0 of device
>> '0000:00:03.0/virtio-balloon'
>> kvm: load of migration failed: Operation not permitted
> 
> Yes that's a known bug; only seems to affect windows guests, and I believe
> doesn't even need to cross versions.
> 
> There's a bunch of fixes that Stefan applied to virtio code
> that I think fix this; I see that he cc'd qemu-stable.
> I think it's 4b7f91ed, but I'm not sure if there are others needed.

thanks for pointing to that commit.

Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into
2.7.0 ?

Greets,
Stefan

> 
> Dave
>>
>> Greets,
>> Stefan
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

On Mon, Nov 14, 2016 at 07:57:47PM +0100, Stefan Priebe - Profihost AG wrote:
> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
> 
> The output of the target qemu process was:
> kvm_apic_post_load: Yeh
> kvm_apic_post_load: Yeh
> kvm_apic_post_load: Yeh
> kvm_apic_post_load: Yeh
> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
> kvm: error while loading state for instance 0x0 of device
> '0000:00:03.0/virtio-balloon'
> kvm: load of migration failed: Operation not permitted

Ladi: I don't remember this.  Does it ring a bell for you?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Ladi Prosek]

Hi,

On Tue, Nov 15, 2016 at 11:37 AM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> Hello,
>
> Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert:
>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>> Hello,
>>>
>>> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
>>> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
>>>
>>> The output of the target qemu process was:
>>> kvm_apic_post_load: Yeh
>>> kvm_apic_post_load: Yeh
>>> kvm_apic_post_load: Yeh
>>> kvm_apic_post_load: Yeh
>>> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
>>> kvm: error while loading state for instance 0x0 of device
>>> '0000:00:03.0/virtio-balloon'
>>> kvm: load of migration failed: Operation not permitted
>>
>> Yes that's a known bug; only seems to affect windows guests, and I believe
>> doesn't even need to cross versions.
>>
>> There's a bunch of fixes that Stefan applied to virtio code
>> that I think fix this; I see that he cc'd qemu-stable.
>> I think it's 4b7f91ed, but I'm not sure if there are others needed.
>
> thanks for pointing to that commit.
>
> Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into
> 2.7.0 ?

I don't believe that 4b7f91ed will help here (no device reset on
migration). We've seen this error with QEMU running without:

  commit 4eae2a657d1ff5ada56eb9b4966eae0eff333b0b
  Author: Ladi Prosek <lprosek@redhat.com>
  Date:   Tue Mar 1 12:14:03 2016 +0100

      balloon: fix segfault and harden the stats queue


Is it possible that the VM has run on such a QEMU, then was
live-migrated to 2.6.2, and then to 2.7.0?

Thanks,
Ladi

> Greets,
> Stefan
>
>>
>> Dave
>>>
>>> Greets,
>>> Stefan
>>>
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>
>

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Stefan Priebe - Profihost AG]

Am 15.11.2016 um 12:07 schrieb Ladi Prosek:
> Hi,
> 
> On Tue, Nov 15, 2016 at 11:37 AM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> Hello,
>>
>> Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert:
>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>>> Hello,
>>>>
>>>> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
>>>> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
>>>>
>>>> The output of the target qemu process was:
>>>> kvm_apic_post_load: Yeh
>>>> kvm_apic_post_load: Yeh
>>>> kvm_apic_post_load: Yeh
>>>> kvm_apic_post_load: Yeh
>>>> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
>>>> kvm: error while loading state for instance 0x0 of device
>>>> '0000:00:03.0/virtio-balloon'
>>>> kvm: load of migration failed: Operation not permitted
>>>
>>> Yes that's a known bug; only seems to affect windows guests, and I believe
>>> doesn't even need to cross versions.
>>>
>>> There's a bunch of fixes that Stefan applied to virtio code
>>> that I think fix this; I see that he cc'd qemu-stable.
>>> I think it's 4b7f91ed, but I'm not sure if there are others needed.
>>
>> thanks for pointing to that commit.
>>
>> Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into
>> 2.7.0 ?
> 
> I don't believe that 4b7f91ed will help here (no device reset on
> migration). We've seen this error with QEMU running without:
> 
>   commit 4eae2a657d1ff5ada56eb9b4966eae0eff333b0b
>   Author: Ladi Prosek <lprosek@redhat.com>
>   Date:   Tue Mar 1 12:14:03 2016 +0100
> 
>       balloon: fix segfault and harden the stats queue
> 
> 
> Is it possible that the VM has run on such a QEMU, then was
> live-migrated to 2.6.2, and then to 2.7.0?

Hi,

yes, it was started under Qemu 2.5.0. Was then migrated to 2.6.2 and
then to 2.7.0.

Greets,
Stefan

> 
> Thanks,
> Ladi
> 
>> Greets,
>> Stefan
>>
>>>
>>> Dave
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>

Re: [Qemu-devel] Qemu 2.6 => Qemu 2.7 migration: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-balloon'

[Ladi Prosek]

On Tue, Nov 15, 2016 at 12:19 PM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> Am 15.11.2016 um 12:07 schrieb Ladi Prosek:
>> Hi,
>>
>> On Tue, Nov 15, 2016 at 11:37 AM, Stefan Priebe - Profihost AG
>> <s.priebe@profihost.ag> wrote:
>>> Hello,
>>>
>>> Am 15.11.2016 um 11:30 schrieb Dr. David Alan Gilbert:
>>>> * Stefan Priebe - Profihost AG (s.priebe@profihost.ag) wrote:
>>>>> Hello,
>>>>>
>>>>> today i did a first live migration from Qemu 2.6.2 to Qemu 2.7.0. The VM
>>>>> is running windows and virtio-balloon and with machine type pc-i440fx-2.5.
>>>>>
>>>>> The output of the target qemu process was:
>>>>> kvm_apic_post_load: Yeh
>>>>> kvm_apic_post_load: Yeh
>>>>> kvm_apic_post_load: Yeh
>>>>> kvm_apic_post_load: Yeh
>>>>> kvm: VQ 2 size 0x80 < last_avail_idx 0x1 - used_idx 0x4
>>>>> kvm: error while loading state for instance 0x0 of device
>>>>> '0000:00:03.0/virtio-balloon'
>>>>> kvm: load of migration failed: Operation not permitted
>>>>
>>>> Yes that's a known bug; only seems to affect windows guests, and I believe
>>>> doesn't even need to cross versions.
>>>>
>>>> There's a bunch of fixes that Stefan applied to virtio code
>>>> that I think fix this; I see that he cc'd qemu-stable.
>>>> I think it's 4b7f91ed, but I'm not sure if there are others needed.
>>>
>>> thanks for pointing to that commit.
>>>
>>> Stefan can you tell me whether it's enough to cherry-pick 4b7f91ed into
>>> 2.7.0 ?
>>
>> I don't believe that 4b7f91ed will help here (no device reset on
>> migration). We've seen this error with QEMU running without:
>>
>>   commit 4eae2a657d1ff5ada56eb9b4966eae0eff333b0b
>>   Author: Ladi Prosek <lprosek@redhat.com>
>>   Date:   Tue Mar 1 12:14:03 2016 +0100
>>
>>       balloon: fix segfault and harden the stats queue
>>
>>
>> Is it possible that the VM has run on such a QEMU, then was
>> live-migrated to 2.6.2, and then to 2.7.0?
>
> Hi,
>
> yes, it was started under Qemu 2.5.0. Was then migrated to 2.6.2 and
> then to 2.7.0.

Thanks. Then I'm afraid there's currently no QEMU fix for the error
you're getting. The virtqueue got into an inconsistent state in 2.5.0
and it carried over to 2.7.0 where the check triggered. As far as I
can tell, the only thing we could do to fix this scenario in 2.7.0
would be to relax the check. It runs before any of the virtio-balloon
code runs.

What makes this specific to Windows guests is that the Windows balloon
driver sends memory stats only if the balloon service (blnsvr.exe) is
running. Older QEMU pulls memory stats unconditionally even if there's
nothing sending them on the guest side.

One possible workaround would be to make sure that blnsvr.exe is
running before the migration. The binary is part of the virtio-win
ISO. I don't think that it guarantees getting around the check but it
may be worth trying.

> Greets,
> Stefan
>
>>
>> Thanks,
>> Ladi
>>
>>> Greets,
>>> Stefan
>>>
>>>>
>>>> Dave
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>> --
>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>
>>>