Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Laine Stump <laine@redhat.com>]

On 10/04/2016 11:45 AM, Alex Williamson wrote:
> On Tue, 4 Oct 2016 15:59:11 +0100
> "Daniel P. Berrange" <berrange@redhat.com> wrote:
>
>> On Mon, Sep 05, 2016 at 06:24:48PM +0200, Laszlo Ersek wrote:
>>> On 09/01/16 15:22, Marcel Apfelbaum wrote:
>>>> +2.3 PCI only hierarchy
>>>> +======================
>>>> +Legacy PCI devices can be plugged into pcie.0 as Integrated Devices or
>>>> +into DMI-PCI bridge. PCI-PCI bridges can be plugged into DMI-PCI bridges
>>>> +and can be nested until a depth of 6-7. DMI-BRIDGES should be plugged
>>>> +only into pcie.0 bus.
>>>> +
>>>> +   pcie.0 bus
>>>> +   ----------------------------------------------
>>>> +        |                            |
>>>> +   -----------               ------------------
>>>> +   | PCI Dev |               | DMI-PCI BRIDGE |
>>>> +   ----------                ------------------
>>>> +                               |            |
>>>> +                        -----------    ------------------
>>>> +                        | PCI Dev |    | PCI-PCI Bridge |
>>>> +                        -----------    ------------------
>>>> +                                         |           |
>>>> +                                  -----------     -----------
>>>> +                                  | PCI Dev |     | PCI Dev |
>>>> +                                  -----------     -----------
>>>
>>> Works for me, but I would again elaborate a little bit on keeping the
>>> hierarchy flat.
>>>
>>> First, in order to preserve compatibility with libvirt's current
>>> behavior, let's not plug a PCI device directly in to the DMI-PCI bridge,
>>> even if that's possible otherwise. Let's just say
>>>
>>> - there should be at most one DMI-PCI bridge (if a legacy PCI hierarchy
>>> is required),
>>
>> Why do you suggest this ? If the guest has multiple NUMA nodes
>> and you're creating a PXB for each NUMA node, then it looks valid
>> to want to have a DMI-PCI bridge attached to each PXB, so you can
>> have legacy PCI devices on each NUMA node, instead of putting them
>> all on the PCI bridge without NUMA affinity.
>
> Seems like this is one of those "generic" vs "specific" device issues.
> We use the DMI-to-PCI bridge as if it were a PCIe-to-PCI bridge, but
> DMI is actually an Intel proprietary interface, the bridge just has the
> same software interface as a PCI bridge.  So while you can use it as a
> generic PCIe-to-PCI bridge, it's at least going to make me cringe every
> time.


If using it this way makes kittens cry or something, then we'd be happy 
to use a generic pcie-to-pci bridge if somebody created one :-)


>
>>> - only PCI-PCI bridges should be plugged into the DMI-PCI bridge,
>>
>> What's the rational for that, as opposed to plugging devices directly
>> into the DMI-PCI bridge which seems to work ?
>
> IIRC, something about hotplug, but from a PCI perspective it doesn't
> make any sense to me either.


At one point Marcel and Michael were discussing the possibility of 
making hotplug work on a dmi-to-pci-bridge. Currently it doesn't even 
work for pci-bridge so (as I think I said in another message just now) 
it is kind of pointless, although when I asked about eliminating use of 
pci-bridge in favor of just using dmi-to-pci-bridge directly, I got lots 
of "no" votes.


>  Same with the restriction from using slot
> 0 on PCI bridges, there's no basis for that except on the root bus.

I tried allowing devices to be plugged into slot 0 of a pci-bridge in 
libvirt - qemu barfed, so I moved the "minSlot" for pci-bridge back up 
to 1. Slot 0 is completely usable on a dmi-to-pci-bridge though (and 
libvirt allows it). At this point, even if qemu enabled using slot 0 of 
a pci-bridge, libvirt wouldn't be able to expose that to users (unless 
the min/max slot of each PCI controller was made visible somewhere via QMP)