Re: [Xen 4.12 bug] HVM/PVH boot confusion

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 30/01/2019 11:25, Roger Pau Monné wrote:
> Adding Anthony who likely knows more about all this since it's closely
> related to QEMU.
>
> On Tue, Jan 29, 2019 at 07:49:51PM +0000, Andrew Cooper wrote:
>> Hello,
>>
>> Given the following vm.cfg file:
>>
>> name="vm"
>> type="hvm"
>>
>> vcpus=4
>> memory=1024
>>
>> firmware_override="/root/xen-syms"
>>
>> kernel="/boot/vmlinuz-4.4-xen"
>> ramdisk="/boot/initrd-4.4.0+10.img"
> I know very little about direct kernel booting with HVM, but I assume
> using firmware_override gets rid of hvmloader and the BIOS/UEFI
> payload plus the option rom to direct boot into a Linux kernel?

Correct.

> So basically the module list is `mod[0] = kernel` and `mod[1] =
> ramdisk`?

I'd expect so.  That was setup I was aiming for.

>> cmdline="console=xen,pv dom0=pv --- earlyprintk=xen"
>>
>> Xen crashes with the following trace:
>>
>> (d15) (XEN) Xen BUG at pvh-boot.c:82
>> (d15) (XEN) ----[ Xen-4.12.0-rc  x86_64  debug=y   Not tainted ]----
>> (d15) (XEN) CPU:    0
>> (d15) (XEN) RIP:    e008:[<ffff82d0804331f2>] pvh_init+0x27d/0x2fe
>> <snip>
>> (d15) (XEN) Xen call trace:
>> (d15) (XEN)    [<ffff82d0804331f2>] pvh_init+0x27d/0x2fe
>> (d15) (XEN)    [<ffff82d080429000>] __start_xen+0x14c/0x28f6
>> (d15) (XEN)    [<ffff82d0802000f3>] __high_start+0x53/0x55
>> (d15) (XEN)
>> (d15) (XEN)
>> (d15) (XEN) ****************************************
>> (d15) (XEN) Panic on CPU 0:
>> (d15) (XEN) Xen BUG at pvh-boot.c:82
>> (d15) (XEN) ****************************************
>>
>> The problem is that Xen is started at its PVH entrypoint (contrary to
>> the instructions in the vm config file), and Xen unconditionally expects
>> RSDP to be passed.
>>
>> There are at least two bugs here.
>>
>> 1) RSDP was a late addition to the PVH boot protocol.  Xen's PVH
>> entrypoint must not mandate its existence, because there are releases of
>> the domain builder which don't provide it.
> Right, I think it should be fine to attempt to boot without a rsdp
> hint, Xen can always resort to scanning the low 1MB in order to
> attempt to find the rsdp.

Agreed.

> The problem here would be that the toolstack is not going to create
> any ACPI tables because it's a HVM guest and thus the toolstack
> expects the firmware to create such tables (hvmloader). Since in the
> above example you skip the firmware, you won't get any ACPI tables at
> all.

This, while true, isn't actually a problem for the case I'm testing.

But yes - I don't expect to be able to substitute Xen for hvmloader in
the general case.  What I didn't expect was for Xen to explode in the
way it did.

>
>> 2) The HVM/PVH boot confusion.  This think this is a still-outstanding
>> bug around the broken assumption that the hvmloader binary speaks the
>> PVH protocol without advertising itself appropriately (I really regret
>> not objecting to those patches before they went in).  At the least, that
>> needs fixing by putting a proper ELF note in hvmloader, and the domain
>> builder needs to be updated to build all PVH-boot-ABI images consistently.
> Hm, booting hvmloader using the hvm_start_info has always been kind of
> a hack, that relied on the toolstack being in full control of the
> firmware and the user not playing with it.

Its doubly a hack because our provided hvmloader binary doesn't
advertise itself as a PVH-capable binary.

> I think part of the problem here also comes from the fact that the
> loader list (xc_dom_loader *first_loader) is shared between
> all the guest types, thus allowing a guest in a hvm container to be
> booted using the pvh entry point.

I think it would help to have a clear separation between the
PVH-boot-ABI (which is used by HVM guests as well), and the concept of a
PVH guest simply being an HVM with less emulation.

The domain builder does need to have some knowledge, even if only for
building the memory map.

>
> I think a proper fix to this mess involves the HVM and the PVH domain
> creation paths being exactly the same, ie: ACPI tables always created
> by the toolstack for example. The only difference between a PVH and a
> HVM guests from the toolstack PoV should be the emulated devices that
> it gets.

This is definitely a good move.  We've got too many different ways of
constructing guests, and there is a lot of unnecessary duplication.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel