Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 7 Sep 2016 11:06:45 +0300
Marcel Apfelbaum <marcel@redhat.com> wrote:

> On 09/07/2016 09:21 AM, Gerd Hoffmann wrote:
> >   Hi,
> >  
> >>>> ports, if that's allowed). For example:
> >>>>
> >>>> -  1-32 ports needed: use root ports only
> >>>>
> >>>> - 33-64 ports needed: use 31 root ports, and one switch with 2-32
> >>>> downstream ports  
> >
> > I expect you rarely need any switches.  You can go multifunction with
> > the pcie root ports.  Which is how physical q35 works too btw, typically
> > the root ports are on slot 1c for intel chipsets:
> >
> > nilsson root ~# lspci -s1c
> > 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
> > Family PCI Express Root Port 1 (rev c4)
> > 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
> > Family PCI Express Root Port 2 (rev c4)
> > 00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
> > Family PCI Express Root Port 3 (rev c4)
> >
> > Root bus has 32 slots, a few are taken (host bridge @ 00.0, lpc+sata @
> > 1f.*, pci bridge @ 1e.0, maybe vga @ 01.0), leaving 28 free slots.  With
> > 8 functions each you can have up to 224 root ports without any switches,
> > and you have not many pci bus numbers left until you hit the 256 busses
> > limit ...
> >  
> 
> Good point, maybe libvirt can avoid adding switches unless the user explicitly
> asked for them. I checked and it a actually works fine in QEMU.
> 
> BTW, when we will implement ARI we can have up to 256 Root Ports on a single slot...

"Root Ports on a single slot"... The entire idea of ARI is that there
is no slot, the slot/function address space is combined into one big
8-bit free-for-all.  Besides, can you do ARI on the root complex?
Typically you need to look at whether the port upstream of a given
device supports ARI to enable, there's no upstream PCI device on the
root complex.  This is the suggestion I gave you for switches, if the
upstream switch port supports ARI then we can have 256 downstream
switch ports (assuming ARI isn't only specific to downstream ports).
Thanks,

Alex