Re: [Qemu-devel] [PATCH RFC] docs: add PCIe devices placement guidelines

[Laine Stump <laine@redhat.com>]

On 10/04/2016 11:40 AM, Laszlo Ersek wrote:
> On 10/04/16 16:59, Daniel P. Berrange wrote:
>> On Mon, Sep 05, 2016 at 06:24:48PM +0200, Laszlo Ersek wrote:
>>> On 09/01/16 15:22, Marcel Apfelbaum wrote:
>>>> +2.3 PCI only hierarchy
>>>> +======================
>>>> +Legacy PCI devices can be plugged into pcie.0 as Integrated Devices or
>>>> +into DMI-PCI bridge. PCI-PCI bridges can be plugged into DMI-PCI bridges
>>>> +and can be nested until a depth of 6-7. DMI-BRIDGES should be plugged
>>>> +only into pcie.0 bus.
>>>> +
>>>> +   pcie.0 bus
>>>> +   ----------------------------------------------
>>>> +        |                            |
>>>> +   -----------               ------------------
>>>> +   | PCI Dev |               | DMI-PCI BRIDGE |
>>>> +   ----------                ------------------
>>>> +                               |            |
>>>> +                        -----------    ------------------
>>>> +                        | PCI Dev |    | PCI-PCI Bridge |
>>>> +                        -----------    ------------------
>>>> +                                         |           |
>>>> +                                  -----------     -----------
>>>> +                                  | PCI Dev |     | PCI Dev |
>>>> +                                  -----------     -----------
>>>
>>> Works for me, but I would again elaborate a little bit on keeping the
>>> hierarchy flat.
>>>
>>> First, in order to preserve compatibility with libvirt's current
>>> behavior, let's not plug a PCI device directly in to the DMI-PCI bridge,
>>> even if that's possible otherwise. Let's just say
>>>
>>> - there should be at most one DMI-PCI bridge (if a legacy PCI hierarchy
>>> is required),
>>
>> Why do you suggest this ? If the guest has multiple NUMA nodes
>> and you're creating a PXB for each NUMA node, then it looks valid
>> to want to have a DMI-PCI bridge attached to each PXB, so you can
>> have legacy PCI devices on each NUMA node, instead of putting them
>> all on the PCI bridge without NUMA affinity.
>
> You are right. I meant the above within one PCI Express root bus.
>
> Small correction to your wording though: you don't want to attach the
> DMI-PCI bridge to the PXB device, but to the extra root bus provided by
> the PXB.

This made me realize something - the root bus on a pxb-pcie controller 
has a single slot and that slot can accept either a pcie-root-port 
(ioh3420) or a dmi-to-pci-bridge. If you want to have both express and 
legacy PCI devices on the same NUMA node, then you would either need to 
create one pxb-pcie for the pcie-root-port and another for the 
dmi-to-pci-bridge, or you would need to put the pcie-root-port and 
dmi-to-pci-bridge onto different functions of the single slot. Should 
the latter work properly?


>
>>
>>> - only PCI-PCI bridges should be plugged into the DMI-PCI bridge,
>>
>> What's the rational for that, as opposed to plugging devices directly
>> into the DMI-PCI bridge which seems to work ?
>
> The rationale is that libvirt used to do it like this.


Nah, that's just the *result* of the rationale that we wanted the 
devices to be hotpluggable. At some later date we learned the hotplug on 
a pci-bridge device doesn't work on a Q35 machine anyway, so it was kind 
of pointless (but we still do it because we hold out hope that hotplug 
of legacy PCI devices into a pci-bridge on Q35 machines will work one day)


> And the rationale
> for *that* is that DMI-PCI bridges cannot accept hotplugged devices,
> while PCI-PCI bridges can.
>
> Technically nothing forbids (AFAICT) cold-plugging PCI devices into
> DMI-PCI bridges, but this document is expressly not just about technical
> constraints -- it's a policy document. We want to simplify / trim the
> supported PCI and PCI Express hierarchies as much as possible.
>
> All valid *high-level* topology goals should be permitted / covered one
> way or another by this document, but in as few ways as possible --
> hopefully only one way. For example, if you read the rest of the thread,
> flat hierarchies are preferred to deeply nested hierarchies, because
> flat ones save on bus numbers

Do they?

>, are easier to setup and understand,
> probably perform better, and don't lose any generality for cold- or hotplug.
>
> Thanks
> Laszlo
>