Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring

["Roger Pau Monné" <roger.pau@citrix.com>]

On Tue, Mar 27, 2018 at 05:42:11AM +1000, Alexey G wrote:
> On Mon, 26 Mar 2018 10:24:38 +0100
> Roger Pau Monné <roger.pau@citrix.com> wrote:
> 
> >On Sat, Mar 24, 2018 at 08:32:44AM +1000, Alexey G wrote:
> [...]
> >> In fact, the emulated chipset (NB+SB combo without supplemental
> >> devices) itself is a small part of required emulation. It's
> >> relatively easy to provide own analogs of for eg. 'mch' and
> >> 'ICH9-LPC' QEMU PCIDevice's, the problem is to glue all remaining
> >> parts together.
> >> 
> >> I assume the final goal in this case is to have only a set of
> >> necessary QEMU PCIDevice's for which we will be providing I/O, MMIO
> >> and PCI conf trapping facilities. Only devices such as rtl8139,
> >> ich9-ahci and few others.
> >> 
> >> Basically, this means a new, chipset-less QEMU machine type.
> >> Well, in theory it is possible with a bit of effort I think. The main
> >> question is where will be the NB/SB/PCIbus emulating part reside in
> >> this case.  
> >
> >Mostly inside of Xen. Of course the IDE/SATA/USB/Ethernet... part of
> >the southbrigde will be emulated by a device model (ie: QEMU).
> >
> >As you mention above, I also took a look and it seems like the amount
> >of registers that we should emulate for Q35 DRAM controller (D0:F0) is
> >fairly minimal based on current QEMU implementation. We could even
> >possibly get away by just emulating PCIEXBAR.
> 
> MCH emulation alone might be not an option. Besides, some
> southbridge-specific features like emulating ACPI PM facilities for
> domain power management (basically, anything at PMBASE) will be
> preferable to implement on Xen side, especially considering the fact
> that ACPI tables are already provided by Xen's libacpi/hvmloader, not
> the device model.

Likely, but AFAICT this is kind of already broken, because PM1a and
TMR is already emulated by Xen at hardcoded values. See
xen/arch/x86/hvm/pmtimer.c.

> I think the feature may require to cover at least the NB+SB
> combination, at least Q35 MCH + ICH9 for start, ideally 82441FX+PIIX4
> as well. Also, Xen should control emulated/PT PCI device placement.

Q35 MCH (D0:F0) it's required in order to trap access to PCIEXBAR.

Could you be more concise about ICH9?

The ICH9 spec contains multiple devices, for example it includes an
ethernet controller and a SATA controller, which we should not emulate
inside of Xen.

> II. (a new feature) Move chipset emulation to Xen directly.
> 
> In this case no separate notification necessary as Xen will be
> emulating the chosen chipset itself. MMCONFIG location will be known
> from own PCIEXBAR emulation.
> 
> QEMU will be used only to emulate a minimal set of unrelated devices
> (eg. storage/network/vga). Less dependency on QEMU overall.
> 
> More freedom to implement some specific features in the future like
> smram support for EFI firmware needs. Chipset remapping (aka reclaim)
> functionality for memory relocation may be implemented under complete
> Xen control, avoiding usage of unsafe add_to_physmap hypercalls.
> 
> In future this will allow to move passthrough-supporting code from QEMU
> (hw/xen/xen-pt*.c) to Xen, merging it with Roger's vpci series.
> This will improve eg. the PT + stubdomain situation a lot -- PCI config
> space accesses for PT devices will be handled in a uniform way without
> Dom0 interaction.
> This particular feature can be implemented for the previous approach as
> well, still it is easier to do when Xen controls the emulated machine
> 
> In general, this is a good long-term direction.
> 
> What this approach will require:
> --------------------------------
> 
> - Changes in QEMU code to support a new chipset-less machine(s). In
>   theory might be possible to implement on top of the "null" machine
>   concept

Not all parts of the chipset should go inside of Xen, ATM I only
foresee Q35 MCH being implemented inside of Xen. So I'm not sure
calling this a chipset-less machine is correct from QEMU PoV.

> - Major changes in Xen code to implement the actual chipset emulation
>   there
> 
> - Changes on the toolstack side as the emulated machine will be
>   selected and used differently
> 
> - Moving passthrough support from QEMU to Xen will likely require to
>   re-divide areas of responsibility for PCI device passthrough between
>   xen-pciback and the hypervisor. It might be more convenient to
>   perform some tasks of xen-pciback in Xen directly

Moving pci-passthough from QEMU to Xen is IMO a separate project, and
by the text you provide I'm not sure how is that related to the Q35
chipset implementation.

> - strong dependency between Xen/libxl/QEMU/etc versions -- any outdated
>   component will be a major problem. Can be resolved by providing some
>   compatibility code

Well, you would only be able to use the Q35 feature with the right
version of the components.

> - longer implementation time
> 
> Risks:
> ------
> 
> - A major architecture change with possible issues encountered during
>   the implementation
> 
> - Moving the emulation of the machine to Xen creates a non-zero risk of
>   introducing a security issue while extending the emulation support
>   further. As all emulation will take place on a most trusted level, any
>   exploitable bug in the chipset emulation code may compromise the
>   whole system
> 
> - there is a risk to encounter some dependency on missing chipset
>   devices in QEMU. Some of QEMU devices (which depend on QEMU chipset
>   devices/properties) might not work without extra patches. In theory
>   this may be addressed by leaving the dummy MCH/LPC/pci-host devices
>   in place while not forwarding any IO/MMIO/PCI conf accesses to them
>   (using simply as compat placeholders)
> 
> - risk of incompatibility with future QEMU versions
> 
> In both cases, for security concerns PCIEXBAR and other MCH registers
> can be made write-once (RO on all further accesses, similar to a
> TXT-locked system).

I think option II is the right way to move forward.

> It is somewhat related to the chipset because memory/MMIO layout
> inconsistency can be solved more, well, naturally on Q35.
> 
> Basically, we have a non-standard MMIO hole layout where the
> start of the high MMIO hole do not match the top of addressable RAM
> (due to invisible ranges of the device model).

But that's a device model issue then? I'm not sure I'm getting what
you mean here.

> Q35 initially have facilities to allow firmware to modify (via
> emulation) or discover such MMIO hole setup which can be used for safe
> MMIO BAR allocation to avoid overlaps with QEMU-owned invisible ranges.

IMO a single entity should be in control of the memory layout, and
that's the toolstack.

Ideally we should not allow the firmware to change the layout at all.
What are specifically the registers that you mention?

> It doesn't really matter which registers to pick for this task, but for
> Q35 this approach is at least consistent with what a real system does
> (PV/PVH people will find this peculiarity pointless I suppose :) ).

Right, but I don't think we aim to emulate a fully complete Q35 MCH or
ICH9 for example, which has tons of registers, not even QEMU is trying
to do that. The main goal is to emulate the registers we know are
required for OSes to work.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel