From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40654) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btz7N-0000EA-M1 for qemu-devel@nongnu.org; Tue, 11 Oct 2016 11:37:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1btz7I-0003Cv-IJ for qemu-devel@nongnu.org; Tue, 11 Oct 2016 11:37:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52686) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1btz7I-0003Ci-8S for qemu-devel@nongnu.org; Tue, 11 Oct 2016 11:37:20 -0400 Message-ID: <1476200234.3672.17.camel@redhat.com> From: Andrea Bolognani Date: Tue, 11 Oct 2016 17:37:14 +0200 In-Reply-To: <73c620b4-bb8d-4bcd-ebd2-d2a95689f2da@redhat.com> References: <1472736127-18137-1-git-send-email-marcel@redhat.com> <6c9103e7-70ad-55c6-5533-d6d9bbaa39b4@redhat.com> <20161004145911.GA2155@redhat.com> <2ea37764-7787-0b1b-5abb-9f8ba12a7080@redhat.com> <2db6a14f-731d-422d-a928-a856360406fd@redhat.com> <547407ca-e8f8-3327-d1c8-8a5c6317f28f@redhat.com> <20161004125229.6ac23806@t450s.home> <1476100967.3672.3.camel@redhat.com> <73c620b4-bb8d-4bcd-ebd2-d2a95689f2da@redhat.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Marcel Apfelbaum , Alex Williamson , Laine Stump Cc: qemu-devel@nongnu.org, Laszlo Ersek , "Daniel P. Berrange" , Peter Maydell , Drew Jones , mst@redhat.com, Gerd Hoffmann On Mon, 2016-10-10 at 17:36 +0300, Marcel Apfelbaum wrote: > > What's the advantage in using ARI to stuff more than eight > > of anything that's not Endpoint Devices in a single slot? > >=C2=A0 > > I mean, if we just fill up all 32 slots in a PCIe Root Bus > > with 8 PCIe Root Ports each we already end up having 256 > > hotpluggable slots[1]. Why would it be preferable to use > > ARI, or even PCIe Switches, instead? >=C2=A0 > What if you need more devices (functions actually) ? >=C2=A0 > If some of the pcie.0 slots are occupied by other Integrated devices > and you need more than 256 functions you can: > (1) Add a PCIe Switch - if you need hot-plug support -an you are pretty= limited >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0by the bus numbers, but it will give= you a few more slots. > (2) Use multi-function devices per root port if you are not interested = in hotplug. >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0In this case ARI will give you up to= 256 devices per Root Port. >=C2=A0 > Now the question is why ARI? Better utilization of the "problematic" > resources like Bus numbers and IO space; all that if you need an insane > number of devices, but we don't judge :). My point is that AIUI ARI is something you only care about for endpoint devices that want to have more than 8 functions. When it comes to controller, there's no advantage that I can think of in having 1 slot with 256 functions as opposed to 32 slots with 8 functions each; if anything, I expect that at least some guest OSs would be quite baffled in finding eg. a network adapter, a SCSI controller and a GPU as separate functions of a single PCI slot. > > [1] The last slot will have to be limited to 7 PCIe Root > >=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Ports if we don't want to run out of bus= numbers >=C2=A0 > I don't follow how this will 'save' us. If all the root ports > are in use and you leave space for one more, what can you do with it? Probably my math is off, but if we can only have 256 PCI buses (0-255) and we plug a PCIe Root Port in each of the 8 functions (0-7) of the 32 ports (0-31) available on the PCIe Root Bus, we end up with =C2=A0 0:00.[0-7] -> [001-008]:0.[0-7] =C2=A0 0:01.[0-7] -> [009-016]:0.[0-7] =C2=A0 0:02.[0-7] -> [017-024]:0.[0-7] =C2=A0 ... =C2=A0 0.30.[0-7] -> [241-248]:0.[0-7] =C2=A0 0.31.[0-7] -> [249-256]:0.[0-7] but 256 is not a valid bus number, so we should skip that last PCIe Root Port and stop at 255. --=C2=A0 Andrea Bolognani / Red Hat / Virtualization