Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Alex Williamson <alex.williamson@redhat.com>]

On Tue, 4 Oct 2016 14:08:45 -0400
Laine Stump <laine@redhat.com> wrote:

> On 10/04/2016 12:43 PM, Laszlo Ersek wrote:
> > On 10/04/16 18:10, Laine Stump wrote:  
> >> On 10/04/2016 11:40 AM, Laszlo Ersek wrote:  
> >>> On 10/04/16 16:59, Daniel P. Berrange wrote:  
> >>>> On Mon, Sep 05, 2016 at 06:24:48PM +0200, Laszlo Ersek wrote:  
> >>> All valid *high-level* topology goals should be permitted / covered one
> >>> way or another by this document, but in as few ways as possible --
> >>> hopefully only one way. For example, if you read the rest of the thread,
> >>> flat hierarchies are preferred to deeply nested hierarchies, because
> >>> flat ones save on bus numbers  
> >>
> >> Do they?  
> >
> > Yes. Nesting implies bridges, and bridges take up bus numbers. For
> > example, in a PCI Express switch, the upstream port of the switch
> > consumes a bus number, with no practical usefulness.  
> 
> I'ts all just idle number games, but what I was thinking of was the 
> difference between plugging  a bunch of root-port+upstream+downstreamxN 
> combos directly into pcie-root (flat), vs. plugging the first into 
> pcie-root, and then subsequent ones into e.g. the last downstream port 
> of the previous set. Take the simplest case of needing 63 hotpluggable 
> slots. In the "flat" case, you have:
> 
>     2 x pcie-root-port
>     2 x pcie-switch-upstream-port
>     63 x pcie-switch-downstream-port
> 
> In the "nested" or "chained" case you have:
> 
>     1 x pcie-root-port
>     1 x pcie-switch-upstream-port
>     32 x pcie-downstream-port
>     1 x pcie-switch-upstream-port
>     32 x pcie-switch-downstream-port

You're not thinking in enough dimensions.  A single root port can host
multiple sub-hierarchies on it's own.  We can have a multi-function
upstream switch, so you can have 8 upstream ports (00.{0-7}).  If we
implemented ARI on the upstream ports, we could have 256 upstream ports
attached to a single root port, but of course then we've run out of
bus numbers before we've even gotten to actual devices buses.

Another option, look at the downstream ports, why do they each need to
be in separate slots?  We have the address space of an entire bus to
work with, so we can also create multi-function downstream ports, which
gives us 256 downstream ports per upstream port.  Oops, we just ran out
of bus numbers again, but at least actual devices can be attached.
Thanks,

Alex


> so you use the same number of PCI controllers.
> 
> Of course if you're talking about the difference between using 
> upstream+downstream vs. just having a bunch of pcie-root-ports directly 
> on pcie-root then you're correct, but only marginally - for 63 
> hotpluggable ports, you would need 63 x pcie-root-port, so a savings of 
> 4 controllers - about 6.5%. (Of course this is all moot since you run 
> out of ioport space after, what, 7 controllers needing it anyway? :-P)
> 
> >
> > IIRC we collectively devised a flat pattern elsewhere in the thread
> > where you could exhaust the 0..255 bus number space such that almost
> > every bridge (= taking up a bus number) would also be capable of
> > accepting a hot-plugged or cold-plugged PCI Express device. That is,
> > practically no wasted bus numbers.
> >
> > Hm.... search this message for "population algorithm":
> >
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg394730.html
> >
> > and then Gerd's big improvement / simplification on it, with multifunction:
> >
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg395437.html
> >
> > In Gerd's scheme, you'd only need only one or two (I'm lazy to count
> > exactly :)) PCI Express switches, to exhaust all bus numbers. Minimal
> > waste due to upstream ports.  
> 
> Yep. And in response to his message, that's what I'm implementing as the 
> default strategy in libvirt :-)
> 
>