[Qemu-devel] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Qemu-devel] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Christian Ehrhardt]

Hi,
migration issues due to rom changes seem to occur over and over in past
years [1], [2],[3],[4],[5].
>From the past I know several workarounds (like just truncating to the
bigger size) but all have various deficiencies.

But OTOH rom's will always change due to fixes in them. And recently I
found one such change [6] that affects the next Ubuntu release and wonder
what the ?right?, well lets say best way to fix it would be.
Current issue:
Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x40000 in != 0x80000:
Invalid argument
Due to efi-virtio.rom growing over 256k due to an update to a newer ipxe
from upstream.

I beg your pardon (for not being educated enough on this yet), but I want
to avoid to go a route that is fixing it in a sub-optimal way and ask for
some guidance. There might be better ways in the modern qemu of today than
there were in the past.
TL;DR I look for the best way (if any) to declare in the new qemu: "I know
that the old size was smaller, let me fix that up on migration".

I'll try on my own as I expect the machine type structures/mechanisms (and
we have Ubuntu specific types that could encapsulate the rom status from
the ipxe package to be coupled with the type) have all that is needed.
Yet me almost randomly trying around there surely isn't the best way to go
- so if there is some existing case or a short hint at what in there might
be the best way to fixup a changed rom size on a migration I'd be very
happy to hear about that.

[1]: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1536331
[2]: https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg01881.html
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1293566
[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1090093
[5]: https://forge.univention.org/bugzilla/show_bug.cgi?id=38877
[6]: https://bugs.launchpad.net/ubuntu/+source/ipxe/+bug/1713490

P.S: As everybody else I don't mind so much on reverse migration to older
releases

-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Re: [Qemu-devel] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Christian Ehrhardt]

Ping - since there wasn't any reply so far - any best practices one could
share?

Let me add a TL;DR:
- bump of ipxe rom versions change the size of virtio-net-pci.rom
- that breaks on migration "Length mismatch"

I'd guess the size of that rom has to be fixed up on the fly, but if that
is really ok and how/where is the question.

Also to +1 on bad things for today - I made this a cross post to libvirt in
case there is one that has done that in the past.


On Mon, Aug 28, 2017 at 4:36 PM, Christian Ehrhardt <
christian.ehrhardt@canonical.com> wrote:

> Hi,
> migration issues due to rom changes seem to occur over and over in past
> years [1], [2],[3],[4],[5].
> From the past I know several workarounds (like just truncating to the
> bigger size) but all have various deficiencies.
>
> But OTOH rom's will always change due to fixes in them. And recently I
> found one such change [6] that affects the next Ubuntu release and wonder
> what the ?right?, well lets say best way to fix it would be.
> Current issue:
> Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x40000 in != 0x80000:
> Invalid argument
> Due to efi-virtio.rom growing over 256k due to an update to a newer ipxe
> from upstream.
>
> I beg your pardon (for not being educated enough on this yet), but I want
> to avoid to go a route that is fixing it in a sub-optimal way and ask for
> some guidance. There might be better ways in the modern qemu of today than
> there were in the past.
> TL;DR I look for the best way (if any) to declare in the new qemu: "I know
> that the old size was smaller, let me fix that up on migration".
>
> I'll try on my own as I expect the machine type structures/mechanisms (and
> we have Ubuntu specific types that could encapsulate the rom status from
> the ipxe package to be coupled with the type) have all that is needed.
> Yet me almost randomly trying around there surely isn't the best way to go
> - so if there is some existing case or a short hint at what in there might
> be the best way to fixup a changed rom size on a migration I'd be very
> happy to hear about that.
>
> [1]: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1536331
> [2]: https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg01881.html
> [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1293566
> [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1090093
> [5]: https://forge.univention.org/bugzilla/show_bug.cgi?id=38877
> [6]: https://bugs.launchpad.net/ubuntu/+source/ipxe/+bug/1713490
>
> P.S: As everybody else I don't mind so much on reverse migration to older
> releases
>
> --
> Christian Ehrhardt
> Software Engineer, Ubuntu Server
> Canonical Ltd
>



-- 
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Re: [Qemu-devel] [libvirt] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Daniel P. Berrange]

On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
> Ping - since there wasn't any reply so far - any best practices one could
> share?
> 
> Let me add a TL;DR:
> - bump of ipxe rom versions change the size of virtio-net-pci.rom
> - that breaks on migration "Length mismatch"
> 
> I'd guess the size of that rom has to be fixed up on the fly, but if that
> is really ok and how/where is the question.

The actual ROM contents will be transferred in the migration stream, so
the fact that the target host has ROMs with different content is not
important. The key thing that matters is that QEMU the target host loads
the ROMs at the same location, so that when the ROM contents is overwritten
with data from the incoming migration scheme, it all ends up at the same
place as it was on the source.

Getting this to happen requires pre-planning when building the ROMs. By
the time you hit the size change during migration it is too late and your
VM is basically doomed. When building you need to add padding. IIUC for
RHEL we artificially increased the size of the seabios and ipxe ROMs to
256k. So when later RHEL updates ship new seabios/ipxe they still fit in
the 256k region previously allowed for.

<tangent>
... QEMU could really benefit from more formal docs around migration to
describe how users / vendors can protect themselves from the many pitfalls
like this...
</tangent>

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [libvirt] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Christian Ehrhardt]

On Thu, Nov 2, 2017 at 4:34 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
>
> On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
> > Ping - since there wasn't any reply so far - any best practices one could
> > share?
> >
> > Let me add a TL;DR:
> > - bump of ipxe rom versions change the size of virtio-net-pci.rom
> > - that breaks on migration "Length mismatch"
> >
> > I'd guess the size of that rom has to be fixed up on the fly, but if that
> > is really ok and how/where is the question.
>
> The actual ROM contents will be transferred in the migration stream, so
> the fact that the target host has ROMs with different content is not
> important. The key thing that matters is that QEMU the target host loads
> the ROMs at the same location, so that when the ROM contents is overwritten
> with data from the incoming migration scheme, it all ends up at the same
> place as it was on the source.


Thanks Daniel for your answer, although you try to kill my remaining
hopes of a better solution :-)
But if the actual ROM content is migrated over anyway "all I'd have to
do" is to make clear that the newer system (qemu) knows that the
incoming rom has a different size than it thinks.
I thought that was what the machine types and their mechanisms were
for, but so far I only scratched like 10% of those - maybe they don't
cover these rom sizes?

>
> Getting this to happen requires pre-planning when building the ROMs. By
> the time you hit the size change during migration it is too late and your
> VM is basically doomed. When building you need to add padding. IIUC for
> RHEL we artificially increased the size of the seabios and ipxe ROMs to
> 256k. So when later RHEL updates ship new seabios/ipxe they still fit in
> the 256k region previously allowed for.


That would have been a good solution to this - if it was done in the past :-)
If I end up forced to draw a line anyway, that is a good way to start
over safe against further size changes.

Re: [Qemu-devel] [libvirt] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Philipp Hahn]

[-- Attachment #1: Type: text/plain, Size: 2554 bytes --]

Hello

Am 03.11.2017 um 08:30 schrieb Christian Ehrhardt:
> On Thu, Nov 2, 2017 at 4:34 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
>>
>> On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
>>> Ping - since there wasn't any reply so far - any best practices one could
>>> share?
>>>
>>> Let me add a TL;DR:
>>> - bump of ipxe rom versions change the size of virtio-net-pci.rom
>>> - that breaks on migration "Length mismatch"
>>>
>>> I'd guess the size of that rom has to be fixed up on the fly, but if that
>>> is really ok and how/where is the question.
>>
>> The actual ROM contents will be transferred in the migration stream, so
>> the fact that the target host has ROMs with different content is not
>> important. The key thing that matters is that QEMU the target host loads
>> the ROMs at the same location, so that when the ROM contents is overwritten
>> with data from the incoming migration scheme, it all ends up at the same
>> place as it was on the source.
> 
> 
> Thanks Daniel for your answer, although you try to kill my remaining
> hopes of a better solution :-)
> But if the actual ROM content is migrated over anyway "all I'd have to
> do" is to make clear that the newer system (qemu) knows that the
> incoming rom has a different size than it thinks.
> I thought that was what the machine types and their mechanisms were
> for, but so far I only scratched like 10% of those - maybe they don't
> cover these rom sizes?

The problem is that libvirt launches a qemu process without knowing
those details.
Later when you try to restore an old snapshot or try to do a live
migration, the newly spawned qemu process must have the same layout to
be able to load the old VMState.

Qemu has gone through several such changes and is worsened by the fact,
that distributions like Debian use their own ROM files, which might be
different from the sizes shipped with Qemu.
For that reason we (Univention) still ship the old ROM images next to
the new ones and carry the attached patch to switch between them based
on the model. Maybe that works for you, too.

One more warning (we experiences): Even if the sizes are the same, live
migration can still fail afterwards: We have observed several cases,
where a migrated VM carried an old version of SeaBIOS, which fails when
you reboot the VM. We traced that down to Qemu implementing a new
feature (SMM), which was not supported by the old SeaBIOS and then got
it wrong after reboot only.

Hope that helps.

Philipp

[-- Attachment #2: 0002-Bug-24702-Rom-file-compatibility.quilt --]
[-- Type: text/plain, Size: 1471 bytes --]

Bug #24702: Fix PXE ROM size mismatch

For pc-0.14 and older use the PXE ROMs from the deprecated etherboot-qemu
package, which have different sizes. Without the patch the virtual PC gets
build with the current ROM sizes. Loading then fails because the stored PCI ROM
BAR size mismatches the configured size.
--- a/hw/pc_piix.c
+++ b/hw/pc_piix.c
@@ -433,6 +433,30 @@ static QEMUMachine pc_machine_v0_15 = {
             .driver   = "virtio-balloon-pci",\
             .property = "event_idx",\
             .value    = "off",\
+        },{\
+            .driver   = "ne2000",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/rtl8029.rom",\
+        },{\
+            .driver   = "rtl8139",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/rtl8139.rom",\
+        },{\
+            .driver   = "e1000",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/e1000-82540em.rom",\
+        },{\
+            .driver   = "pcnet",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/pcnet32.rom",\
+        },{\
+            .driver   = "ne2k-isa",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/ne.rom",\
+        },{\
+            .driver   = "virtio-net-pci",\
+            .property = "romfile",\
+            .value   = "/usr/lib/etherboot/virtio-net.rom",\
         }
 
 static QEMUMachine pc_machine_v0_14 = {