Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring

[Paul Durrant <Paul.Durrant@citrix.com>]

> -----Original Message-----
> From: Alexey G [mailto:x1917x@gmail.com]
> Sent: 21 March 2018 16:57
> To: Roger Pau Monne <roger.pau@citrix.com>
> Cc: xen-devel@lists.xenproject.org; Andrew Cooper
> <Andrew.Cooper3@citrix.com>; Ian Jackson <Ian.Jackson@citrix.com>; Jan
> Beulich <jbeulich@suse.com>; Wei Liu <wei.liu2@citrix.com>; Paul Durrant
> <Paul.Durrant@citrix.com>; Anthony Perard <anthony.perard@citrix.com>;
> Stefano Stabellini <sstabellini@kernel.org>
> Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG
> area in the MMIO hole + minor code refactoring
> 
> On Wed, 21 Mar 2018 15:20:17 +0000
> Roger Pau Monné <roger.pau@citrix.com> wrote:
> 
> >On Thu, Mar 22, 2018 at 12:25:40AM +1000, Alexey G wrote:
> >> Roger, Paul,
> >>
> >> Here is what you suggest, just to clarify:
> >>
> >> 1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell
> >> Xen where QEMU emulates machine's MMCONFIG (chipset-specific
> >> emulation of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely
> >> on this information to know to which PCI device the address within
> >> MMCONFIG belong.
> >>
> >> 2. Xen will trap this area + remap its trapping to other address if
> >> QEMU will inform Xen about emulated PCIEXBAR value change
> >>
> >> 3. Every MMIO access to the current MMCONFIG range will be converted
> >> into BDF first (by offset within this range, knowing where the range
> >> is)
> >>
> >> 4. Target device model is selected using calculated BDF
> >>
> >> 5. MMIO read/write accesses are converted into PCI config space
> >> ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At
> >> this point ioreq structure allows to specify extended PCI conf offset
> >> (12-bit), so it will fit into PCI conf ioreq. For now let's assume
> >> that eg. a 64-bit memory operation is either aborted or workarounded
> >> by splitting this operation into multiple PCI conf ioreqs.
> >
> >Why can't you just set size = 8 in that case in the ioreq?
> >
> >QEMU should then reject those if the chipset doesn't support 64bit
> >accesses. I cannot find in the spec any mention of whether this
> >chipset supports 64bit MCFG accesses, and according to the PCIe spec
> >64bit accesses to MCFG should not be used unless the chipset is known
> >to handle them correctly.
> Yes, uint64_t should be enough in this particular case in fact, though
> memory nature of MMCONFIG accesses might still require to provide
> specific handling.
> 
> 
> All right then, so it will be a dmop/hypercall to tell Xen where to
> trap MMIO accesses to MMCONFIG as you propose.
> 
> The primary device model (QEMU) will be emulating chipset-specific
> PCIEXBAR/etc and issuing this new dmop to tell Xen which area it needs
> to trap for MMIO MMCONFIG acceses. It's basically what
> map_io_range_to_ioreq_server does currently, but I guess a new dedicated
> dmop/hypercall is bearable.
> 
> >> 6. PCI conf read/write ioreqs are sent to the chosen device model
> >>
> >> 7. QEMU receive MMCONFIG memory reads/writes as PCI conf
> reads/writes
> >>
> >> 8. As these MMCONFIG PCI conf reads occur out of context (just
> >> address/len/data without any emulated device attached to it),
> >> xen-hvm.c should employ special logic to make it QEMU-friendly --
> >> eg. right now it sends received PCI conf access into (emulated by
> >> QEMU) CF8h/CFCh ports.
> >> There is a real problem to embed these "naked" accesses into QEMU
> >> infrastructure, workarounds are required. BTW, find_primary_bus() was
> >> dropped from QEMU code -- it could've been useful here. Let's assume
> >> some workaround is employed (like storing a required object pointers
> >> in global variables for later use in xen-hvm.c)
> >
> >That seems like a minor nit, but why not just use
> >address_space_{read/write} to replay the MCFG accesses as memory
> >read/writes?
> 
> Well, this might work actually. Although the overall scenario will be
> overcomplicated a bit for _PCI_CONFIG ioreqs. Here is how it will look:
> 
> QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new
> MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone
> is
> accessing this area -> Xen intercepts this MMIO access
> 
> But here's what happens next:
> 
> Xen translates MMIO access into PCI_CONFIG and sends it to DM ->
> DM receives _PCI_CONFIG ioreq -> DM translates BDF/addr info back to
> the offset in emulated MMCONFIG range -> DM calls
> address_space_read/write to trigger MMIO emulation
> 

That would only be true of a dm that cannot handle PCI config ioreqs directly.

  Paul

> I tnink some parts of this equation can be collapsed, isn't it?
> 
> Above scenario makes it obvious that at least for QEMU the MMIO->PCI
> conf translation is a redundant step. Why not to allow specifying for DM
> whether it prefers to receive MMCONFIG accesses as native (MMIO ones)
> or as translated PCI conf ioreqs? We can still route either ioreq
> type to multiple device emulators accordingly.
> 
> This will be the most universal and consistent approach -- either _COPY
> or _PCI_CONFIG-type ioreqs can be sent to DM, whatever it likes more.
> 
> >> 9. Existing MMCONFIG-handling code in QEMU will be unused in this
> >> scenario
> >
> >If you replay the read/write I don't think so. In any case this is
> >irrelevant. QEMU CPU emulation code is also unused when running under
> >Xen.
> >
> >> 10. All this needed primarily to make the specific "Multiple device
> >> emulators" feature to work (XenGT was mentioned as its user) on Q35
> >> with MMCONFIG.
> >>
> >> Anything wrong/missing here?
> >
> >I think that's correct.
> >
> >Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel