Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Marcel Apfelbaum <marcel@redhat.com>]

On 09/07/2016 07:08 PM, Alex Williamson wrote:
> On Wed, 7 Sep 2016 11:06:45 +0300
> Marcel Apfelbaum <marcel@redhat.com> wrote:
>
>> On 09/07/2016 09:21 AM, Gerd Hoffmann wrote:
>>>   Hi,
>>>
>>>>>> ports, if that's allowed). For example:
>>>>>>
>>>>>> -  1-32 ports needed: use root ports only
>>>>>>
>>>>>> - 33-64 ports needed: use 31 root ports, and one switch with 2-32
>>>>>> downstream ports
>>>
>>> I expect you rarely need any switches.  You can go multifunction with
>>> the pcie root ports.  Which is how physical q35 works too btw, typically
>>> the root ports are on slot 1c for intel chipsets:
>>>
>>> nilsson root ~# lspci -s1c
>>> 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
>>> Family PCI Express Root Port 1 (rev c4)
>>> 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
>>> Family PCI Express Root Port 2 (rev c4)
>>> 00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
>>> Family PCI Express Root Port 3 (rev c4)
>>>
>>> Root bus has 32 slots, a few are taken (host bridge @ 00.0, lpc+sata @
>>> 1f.*, pci bridge @ 1e.0, maybe vga @ 01.0), leaving 28 free slots.  With
>>> 8 functions each you can have up to 224 root ports without any switches,
>>> and you have not many pci bus numbers left until you hit the 256 busses
>>> limit ...
>>>
>>
>> Good point, maybe libvirt can avoid adding switches unless the user explicitly
>> asked for them. I checked and it a actually works fine in QEMU.
>>
>> BTW, when we will implement ARI we can have up to 256 Root Ports on a single slot...
>
> "Root Ports on a single slot"... The entire idea of ARI is that there
> is no slot, the slot/function address space is combined into one big
> 8-bit free-for-all.  Besides, can you do ARI on the root complex?

No, we can't :(
Indeed, for the Root Complex bus we need the (bus:)dev:fn tuple, because we can
have multiple devices plugged in. Thanks for the correction.

Thanks,
Marcel

> Typically you need to look at whether the port upstream of a given
> device supports ARI to enable, there's no upstream PCI device on the
> root complex.  This is the suggestion I gave you for switches, if the
> upstream switch port supports ARI then we can have 256 downstream
> switch ports (assuming ARI isn't only specific to downstream ports).
> Thanks,
>
> Alex
>