On Tue, Dec 13, 2016 at 09:15:37AM -0600, Benjamin Herrenschmidt wrote:
> On Tue, 2016-12-13 at 14:25 +0200, Marcel Apfelbaum wrote:
> > > > Hrm, the suggestion of providing both a vanilla-PCI and PCI-E host
> > > > bridge came up before.  I think one of us spotted a problem with that,
> > > > but I don't recall what it was now.  I guess one is how libvirt would
> > > > map it's stupid-fake-domain-numbers to which root bus to use.
> > 
> > This would be a weird configuration, I never heard of something like that
> > on a bare metal machine, but I never worked on pseries, who knows...
> 
> That's the thing though, it's *not* bare metal ;-)
> 
> On bare metal, we use the "powernv" platform for which we haven't yet upstreamed
> the PCIE support, but there we have real PCIe root ports with all what's exepcted
> on them.
> 
> They come on physically different PHB, one root complex per PHB, but they are
> "proper" PCIe. Hotplug is a bit weird still because it has to go through some
> FW interactions as the HW doesn't use the PCIe standard slot control bits (and
> our qemu model doesn't handle it yet) but otherwise it's standard.
> 
> "pseries" on the other hand is a paravirtualized platform. It's the environment
> presented to guests by our propritary hypervisor (PowerVM, aka pHyp) and
> KVM/qemu. I think there's a public spec these days but to cut the story short,
> it doesn't expose PCIe "properly" for bad historical reasons.
> 
> It tends to hide the parent, ie, the root port for slots connected directly to
> a PHB or the entire hierarchy from the root to (and including) the downstream
> switch port for slots under as switch.
> 
> So you end up with PCIe devices devoid of a proper "parent" which is a bit
> weird.
> 
> Hotplug is implemented using specific firmware interfaces that are identical
> with pre-PCIe (ie PCI/PCI-X) systems. The FW has interface for supporting 3
> kinds of hotplug:
> 
>    - Entire PHBs
>    - P2P Bridges
>    - Individual devices on an existing PHB or P2P bridge
> 
> In practice these days mostly the first one is used for everything under pHyp,
> though I think we have a mix of 1 and 3 for KVM/qemu.

Alas, no, we only have 3 on KVM/qemu.  Well, possibly you could do (2)
by plugging a bridge device using (3) then a device onto the bridge.
Whole PHB hotplug has never been implemented for qemu/KVM.. but we
probably want to.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson