From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41026) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bhjYa-0001va-6p for qemu-devel@nongnu.org; Wed, 07 Sep 2016 16:34:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bhjYW-0002RY-6w for qemu-devel@nongnu.org; Wed, 07 Sep 2016 16:34:52 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45646) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bhjYV-0002Qv-UK for qemu-devel@nongnu.org; Wed, 07 Sep 2016 16:34:48 -0400 References: <1472736127-18137-1-git-send-email-marcel@redhat.com> <6c9103e7-70ad-55c6-5533-d6d9bbaa39b4@redhat.com> <739af9d6-2382-c4bd-679d-87b153124491@redhat.com> <1473229310.28663.14.camel@redhat.com> <3ed4337a-f53a-a148-88d6-a3d35ffeca79@redhat.com> From: Laine Stump Message-ID: Date: Wed, 7 Sep 2016 16:34:44 -0400 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: Marcel Apfelbaum , Peter Maydell , Drew Jones , mst@redhat.com, Andrea Bolognani , Alex Williamson , Gerd Hoffmann , Laszlo Ersek On 09/07/2016 03:39 PM, Marcel Apfelbaum wrote: > On 09/07/2016 08:55 PM, Laine Stump wrote: >> On 09/07/2016 04:06 AM, Marcel Apfelbaum wrote: >>> On 09/07/2016 09:21 AM, Gerd Hoffmann wrote: >>>> Hi, >>>> >>>>>>> ports, if that's allowed). For example: >>>>>>> >>>>>>> - 1-32 ports needed: use root ports only >>>>>>> >>>>>>> - 33-64 ports needed: use 31 root ports, and one switch with 2-32 >>>>>>> downstream ports >>>> >>>> I expect you rarely need any switches. You can go multifunction with >>>> the pcie root ports. Which is how physical q35 works too btw, >>>> typically >>>> the root ports are on slot 1c for intel chipsets: >>>> >>>> nilsson root ~# lspci -s1c >>>> 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset >>>> Family PCI Express Root Port 1 (rev c4) >>>> 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset >>>> Family PCI Express Root Port 2 (rev c4) >>>> 00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset >>>> Family PCI Express Root Port 3 (rev c4) >>>> >>>> Root bus has 32 slots, a few are taken (host bridge @ 00.0, lpc+sata @ >>>> 1f.*, pci bridge @ 1e.0, maybe vga @ 01.0), leaving 28 free slots. >>>> With >>>> 8 functions each you can have up to 224 root ports without any >>>> switches, >>>> and you have not many pci bus numbers left until you hit the 256 busses >>>> limit ... >>>> >>> >>> Good point, maybe libvirt can avoid adding switches unless the user >>> explicitly >>> asked for them. I checked and it a actually works fine in QEMU. >> >> I'm just now writing the code that auto-adds *-ports as they are >> needed, and doing it this way simplifies it *immensely*. >> >> When I had to think about the possibility of needing >> upstream/downstream switches, as an endpoint device was added, I would >> need to check if a (root|downstream)-port was available and if not I >> might >> be able to just add a root-port, or I might have to add a >> downstream-port; if the only option was a downstream port, then *that* >> might require adding a new *upstream* port. >> >> If I can limit libvirt to only auto-adding root-ports (and if there is >> no downside to putting multiple root ports on a single root bus port), >> then I just need to find an empty function of an empty >> slot on the root bus, add a root-port, and I'm done (and since 224 is >> *a lot*, I think at least for now it's okay to punt once they get past >> that point). >> >> So, *is* there any downside to doing this? >> > > No downside I can think of. > Just be sure to emphasize the auto-add mechanism stops at 'x' devices. > If the user needs more, > he should manually add switches and manually assign the devices to the > Downstream Ports. Actually, just the former - once the downstream ports are added, they'll automatically be used for endpoint devices (and even new upstream ports) as needed.