Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Laine Stump <laine@redhat.com>]

On 09/07/2016 04:06 AM, Marcel Apfelbaum wrote:
> On 09/07/2016 09:21 AM, Gerd Hoffmann wrote:
>>   Hi,
>>
>>>>> ports, if that's allowed). For example:
>>>>>
>>>>> -  1-32 ports needed: use root ports only
>>>>>
>>>>> - 33-64 ports needed: use 31 root ports, and one switch with 2-32
>>>>> downstream ports
>>
>> I expect you rarely need any switches.  You can go multifunction with
>> the pcie root ports.  Which is how physical q35 works too btw, typically
>> the root ports are on slot 1c for intel chipsets:
>>
>> nilsson root ~# lspci -s1c
>> 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
>> Family PCI Express Root Port 1 (rev c4)
>> 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
>> Family PCI Express Root Port 2 (rev c4)
>> 00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
>> Family PCI Express Root Port 3 (rev c4)
>>
>> Root bus has 32 slots, a few are taken (host bridge @ 00.0, lpc+sata @
>> 1f.*, pci bridge @ 1e.0, maybe vga @ 01.0), leaving 28 free slots.  With
>> 8 functions each you can have up to 224 root ports without any switches,
>> and you have not many pci bus numbers left until you hit the 256 busses
>> limit ...
>>
>
> Good point, maybe libvirt can avoid adding switches unless the user
> explicitly
> asked for them. I checked and it a actually works fine in QEMU.

I'm just now writing the code that auto-adds *-ports as they are needed, 
and doing it this way simplifies it *immensely*.

When I had to think about the possibility of needing upstream/downstream 
switches, as an endpoint device was added, I would need to check if a 
(root|downstream)-port was available and if not I might be able to just 
add a root-port, or I might have to add a downstream-port; if the only 
option was a downstream port, then *that* might require adding a new 
*upstream* port.

If I can limit libvirt to only auto-adding root-ports (and if there is 
no downside to putting multiple root ports on a single root bus port), 
then I just need to find an empty function of an empty slot on the root 
bus, add a root-port, and I'm done (and since 224 is *a lot*, I think at 
least for now it's okay to punt once they get past that point).

So, *is* there any downside to doing this?