From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48771) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1brUpY-00079S-MQ for qemu-devel@nongnu.org; Tue, 04 Oct 2016 14:52:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1brUpP-0003bF-EX for qemu-devel@nongnu.org; Tue, 04 Oct 2016 14:52:44 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50162) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1brUpM-0003aG-RG for qemu-devel@nongnu.org; Tue, 04 Oct 2016 14:52:35 -0400 Date: Tue, 4 Oct 2016 12:52:29 -0600 From: Alex Williamson Message-ID: <20161004125229.6ac23806@t450s.home> In-Reply-To: <547407ca-e8f8-3327-d1c8-8a5c6317f28f@redhat.com> References: <1472736127-18137-1-git-send-email-marcel@redhat.com> <6c9103e7-70ad-55c6-5533-d6d9bbaa39b4@redhat.com> <20161004145911.GA2155@redhat.com> <2ea37764-7787-0b1b-5abb-9f8ba12a7080@redhat.com> <2db6a14f-731d-422d-a928-a856360406fd@redhat.com> <547407ca-e8f8-3327-d1c8-8a5c6317f28f@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Laine Stump Cc: qemu-devel@nongnu.org, Laszlo Ersek , "Daniel P. Berrange" , Marcel Apfelbaum , Peter Maydell , Drew Jones , mst@redhat.com, Andrea Bolognani , Gerd Hoffmann On Tue, 4 Oct 2016 14:08:45 -0400 Laine Stump wrote: > On 10/04/2016 12:43 PM, Laszlo Ersek wrote: > > On 10/04/16 18:10, Laine Stump wrote: > >> On 10/04/2016 11:40 AM, Laszlo Ersek wrote: > >>> On 10/04/16 16:59, Daniel P. Berrange wrote: > >>>> On Mon, Sep 05, 2016 at 06:24:48PM +0200, Laszlo Ersek wrote: > >>> All valid *high-level* topology goals should be permitted / covered one > >>> way or another by this document, but in as few ways as possible -- > >>> hopefully only one way. For example, if you read the rest of the thread, > >>> flat hierarchies are preferred to deeply nested hierarchies, because > >>> flat ones save on bus numbers > >> > >> Do they? > > > > Yes. Nesting implies bridges, and bridges take up bus numbers. For > > example, in a PCI Express switch, the upstream port of the switch > > consumes a bus number, with no practical usefulness. > > I'ts all just idle number games, but what I was thinking of was the > difference between plugging a bunch of root-port+upstream+downstreamxN > combos directly into pcie-root (flat), vs. plugging the first into > pcie-root, and then subsequent ones into e.g. the last downstream port > of the previous set. Take the simplest case of needing 63 hotpluggable > slots. In the "flat" case, you have: > > 2 x pcie-root-port > 2 x pcie-switch-upstream-port > 63 x pcie-switch-downstream-port > > In the "nested" or "chained" case you have: > > 1 x pcie-root-port > 1 x pcie-switch-upstream-port > 32 x pcie-downstream-port > 1 x pcie-switch-upstream-port > 32 x pcie-switch-downstream-port You're not thinking in enough dimensions. A single root port can host multiple sub-hierarchies on it's own. We can have a multi-function upstream switch, so you can have 8 upstream ports (00.{0-7}). If we implemented ARI on the upstream ports, we could have 256 upstream ports attached to a single root port, but of course then we've run out of bus numbers before we've even gotten to actual devices buses. Another option, look at the downstream ports, why do they each need to be in separate slots? We have the address space of an entire bus to work with, so we can also create multi-function downstream ports, which gives us 256 downstream ports per upstream port. Oops, we just ran out of bus numbers again, but at least actual devices can be attached. Thanks, Alex > so you use the same number of PCI controllers. > > Of course if you're talking about the difference between using > upstream+downstream vs. just having a bunch of pcie-root-ports directly > on pcie-root then you're correct, but only marginally - for 63 > hotpluggable ports, you would need 63 x pcie-root-port, so a savings of > 4 controllers - about 6.5%. (Of course this is all moot since you run > out of ioport space after, what, 7 controllers needing it anyway? :-P) > > > > > IIRC we collectively devised a flat pattern elsewhere in the thread > > where you could exhaust the 0..255 bus number space such that almost > > every bridge (= taking up a bus number) would also be capable of > > accepting a hot-plugged or cold-plugged PCI Express device. That is, > > practically no wasted bus numbers. > > > > Hm.... search this message for "population algorithm": > > > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg394730.html > > > > and then Gerd's big improvement / simplification on it, with multifunction: > > > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg395437.html > > > > In Gerd's scheme, you'd only need only one or two (I'm lazy to count > > exactly :)) PCI Express switches, to exhaust all bus numbers. Minimal > > waste due to upstream ports. > > Yep. And in response to his message, that's what I'm implementing as the > default strategy in libvirt :-) > >