Re: [PATCH] xen/mcfg: Call PHYSDEVOP_pci_mmcfg_reserved before PCI enumeration

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Sep 22, 2015 at 08:12:56AM -0600, Jan Beulich wrote:
> >>> On 22.09.15 at 16:09, <konrad.wilk@oracle.com> wrote:
> > On Tue, Sep 22, 2015 at 07:57:29AM -0600, Jan Beulich wrote:
> >> >>> On 22.09.15 at 15:36, <konrad.wilk@oracle.com> wrote:
> >> > The best I could come up with is to do two loops:
> >> >  1) for 0:0:0 -> ff:ff:ff call PHYSDEVOP_pci_device_remove
> >> >     (so blow away what Xen has for its PCI devices.. except for the AMD 
> >> > IOMMU)
> >> >  2) list_for_each_pci_device PHYSDEVOP_pci_device_add (or other variants
> >> >     if VF).
> >> > 
> >> > But that is just a hack working around the Linux code.
> >> 
> >> The price of being non-intrusive on the Linux side. The above would
> >> seem okay to me for the (rare?) reassign case. (Not sure why you
> >> mean to exclude the AMD IOMMU there.)
> > 
> > I think we hide it altogether from the dom0 - so when it does the
> > bus reassignment it does not see the AMD IOMMU.
> > 
> > My concern was that the bus number of the AMD IOMMU device may
> > overlap with other bus numbers - which got moved as a result
> > of the bridge expanding its subordinate bus numbers.
> > 
> > In other words, the AMD IOMMU may have been at bus 0xb, the
> > bridge in question at 0x01, with an PCI device at 0x02;
> > some devices between 0x03 down to 0x0a.
> > 
> > The bridge would now cover 0x01->0x04 (with the PCI device
> > still at 0x02). But the devices at the old 0x03->0x0a have now
> > been shifted to 0x04->0x0b. And the AMD IOMMU is at 0x0c.
> > 
> > But Xen has no clue about this and "loses" the AMD IOMMU and
> > starts writting configuration data to whatever poor device used to
> > be at bus 0x0b.
> 
> And how would that issue be solved by leaving that device alone?

If it was exposed to dom0, it could make an PHYSDEVOP_pci_device_add
to add it back to Xen (as a new device at bus 0x0c).

But Xen wouldn't know what to do with it - it would still think that
the AMD IOMMU is at bus 0x0b.

I believe I am going on a tangent here - that is the MMCFG
conversation is about the extended configuration registers and how
to make Xen aware of them such that when the VF is added Xen will
be able to validate them. Ed's idea of making the MMCFG hypercall
when the PCI notification (with your suggestions) should take care
of it (I hope).

The issue I am waxing on about is just Linux reassigning the
PCI devices and what are some of the repercussions that stem from that.

I should start a new thread about it and propose some prototype
patch, once my TODO queue is a bit sane.


> 
> Jan
>