Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring

[Alexey G <x1917x@gmail.com>]

On Wed, 21 Mar 2018 15:20:17 +0000
Roger Pau Monné <roger.pau@citrix.com> wrote:

>On Thu, Mar 22, 2018 at 12:25:40AM +1000, Alexey G wrote:
>> Roger, Paul,
>> 
>> Here is what you suggest, just to clarify:
>> 
>> 1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell
>> Xen where QEMU emulates machine's MMCONFIG (chipset-specific
>> emulation of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely
>> on this information to know to which PCI device the address within
>> MMCONFIG belong.
>> 
>> 2. Xen will trap this area + remap its trapping to other address if
>> QEMU will inform Xen about emulated PCIEXBAR value change
>> 
>> 3. Every MMIO access to the current MMCONFIG range will be converted
>> into BDF first (by offset within this range, knowing where the range
>> is)
>> 
>> 4. Target device model is selected using calculated BDF
>> 
>> 5. MMIO read/write accesses are converted into PCI config space
>> ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At
>> this point ioreq structure allows to specify extended PCI conf offset
>> (12-bit), so it will fit into PCI conf ioreq. For now let's assume
>> that eg. a 64-bit memory operation is either aborted or workarounded
>> by splitting this operation into multiple PCI conf ioreqs.  
>
>Why can't you just set size = 8 in that case in the ioreq?
>
>QEMU should then reject those if the chipset doesn't support 64bit
>accesses. I cannot find in the spec any mention of whether this
>chipset supports 64bit MCFG accesses, and according to the PCIe spec
>64bit accesses to MCFG should not be used unless the chipset is known
>to handle them correctly.
Yes, uint64_t should be enough in this particular case in fact, though
memory nature of MMCONFIG accesses might still require to provide
specific handling.


All right then, so it will be a dmop/hypercall to tell Xen where to
trap MMIO accesses to MMCONFIG as you propose.

The primary device model (QEMU) will be emulating chipset-specific
PCIEXBAR/etc and issuing this new dmop to tell Xen which area it needs
to trap for MMIO MMCONFIG acceses. It's basically what
map_io_range_to_ioreq_server does currently, but I guess a new dedicated
dmop/hypercall is bearable.

>> 6. PCI conf read/write ioreqs are sent to the chosen device model
>> 
>> 7. QEMU receive MMCONFIG memory reads/writes as PCI conf reads/writes
>> 
>> 8. As these MMCONFIG PCI conf reads occur out of context (just
>> address/len/data without any emulated device attached to it),
>> xen-hvm.c should employ special logic to make it QEMU-friendly --
>> eg. right now it sends received PCI conf access into (emulated by
>> QEMU) CF8h/CFCh ports.
>> There is a real problem to embed these "naked" accesses into QEMU
>> infrastructure, workarounds are required. BTW, find_primary_bus() was
>> dropped from QEMU code -- it could've been useful here. Let's assume
>> some workaround is employed (like storing a required object pointers
>> in global variables for later use in xen-hvm.c)  
>
>That seems like a minor nit, but why not just use
>address_space_{read/write} to replay the MCFG accesses as memory
>read/writes?

Well, this might work actually. Although the overall scenario will be
overcomplicated a bit for _PCI_CONFIG ioreqs. Here is how it will look:

QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new
MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone is
accessing this area -> Xen intercepts this MMIO access

But here's what happens next:

Xen translates MMIO access into PCI_CONFIG and sends it to DM ->
DM receives _PCI_CONFIG ioreq -> DM translates BDF/addr info back to
the offset in emulated MMCONFIG range -> DM calls
address_space_read/write to trigger MMIO emulation

I tnink some parts of this equation can be collapsed, isn't it?

Above scenario makes it obvious that at least for QEMU the MMIO->PCI
conf translation is a redundant step. Why not to allow specifying for DM
whether it prefers to receive MMCONFIG accesses as native (MMIO ones)
or as translated PCI conf ioreqs? We can still route either ioreq
type to multiple device emulators accordingly.

This will be the most universal and consistent approach -- either _COPY
or _PCI_CONFIG-type ioreqs can be sent to DM, whatever it likes more.

>> 9. Existing MMCONFIG-handling code in QEMU will be unused in this
>> scenario  
>
>If you replay the read/write I don't think so. In any case this is
>irrelevant. QEMU CPU emulation code is also unused when running under
>Xen.
>
>> 10. All this needed primarily to make the specific "Multiple device
>> emulators" feature to work (XenGT was mentioned as its user) on Q35
>> with MMCONFIG.
>> 
>> Anything wrong/missing here?  
>
>I think that's correct.
>
>Thanks, Roger.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel