Peer bridge fixup issue under multiple pci domain

Peer bridge fixup issue under multiple pci domain

[Zihan Yang]

Hi all,

I'm trying to use multiple pci domain in qemu q35, but I find there
might be some issues in peer bridge fixup.

In short, pcibios_fixup_peer_bridges function assumes only one pci
domain (0) by default. This is OK when as qemu by default uses only
one pci domain too. However, if I add another host bridge which is
put into pci domain 1 by using _SEG, and a pcie_pci_bridge is attached
to the bus 1 under this new pci domain 1 rather than domain 0, the
kernel will recognize the bus 01 differently.

More specifically, pcibios_fixup_peer_bridges only reads all the buses
under domain 0 but it can read the pci bus 01 in pci domain 1 and treat
it as a peer bus of 0000:00. The consequence is this 01 bus is recognized
as 0000:01, but it should have been recognized as 0001:01.

The host bus 0001:00 can be recognized so I guess pcibios_fixup_peer_bridges
needs updating to take care of multiple domains? Or is it just an bios issue?
I'm not quite sure and I'm open to any suggestions.

Thanks,
Ziahn

Re: Peer bridge fixup issue under multiple pci domain

[Bjorn Helgaas]

[+cc EDAC folks, LKML]

On Sat, Aug 25, 2018 at 10:58:57PM +0800, Zihan Yang wrote:
> Hi all,
> 
> I'm trying to use multiple pci domain in qemu q35, but I find there
> might be some issues in peer bridge fixup.
> 
> In short, pcibios_fixup_peer_bridges function assumes only one pci
> domain (0) by default. This is OK when as qemu by default uses only
> one pci domain too. However, if I add another host bridge which is
> put into pci domain 1 by using _SEG, and a pcie_pci_bridge is attached
> to the bus 1 under this new pci domain 1 rather than domain 0, the
> kernel will recognize the bus 01 differently.
> 
> More specifically, pcibios_fixup_peer_bridges only reads all the buses
> under domain 0 but it can read the pci bus 01 in pci domain 1 and treat
> it as a peer bus of 0000:00. The consequence is this 01 bus is recognized
> as 0000:01, but it should have been recognized as 0001:01.
> 
> The host bus 0001:00 can be recognized so I guess pcibios_fixup_peer_bridges
> needs updating to take care of multiple domains? Or is it just an bios issue?
> I'm not quite sure and I'm open to any suggestions.

Is there something that actually does not work, or is this just a
concern that the code looks wrong?

pcibios_fixup_peer_bridges() is ancient history from before x86 used
the ACPI namespace to discover host bridges.  It blindly probes for
devices on buses 0-255, but as you say, only in domain 0.

Using multiple PCI domains really requires ACPI support so we know
what the other domains are (_SEG) and how to access their config space
(MCFG).  When we do have ACPI support in the platform and the kernel,
drivers/acpi/pci_root.c discovers all the host bridges in all domains
via PNP0A03 or PNP0A08 devices in the ACPI namespace, and in most
cases pcibios_fixup_peer_bridges() will do nothing.

However, there *are* systems where the firmware does not expose all
host bridges and in those cases, pcibios_fixup_peer_bridges() can be a
problem.  For example, Intel processors often have management devices
on bus 7f or ff.  If the ACPI namespace doesn't have a host bridge to
those buses, pci_root.c won't find them, but
pcibios_fixup_peer_bridges() *will*.

This leads to several problems.  Here's a dmesg sample from [1]
(found by googling for 'dmesg log "PCI: discovered peer bus ff"'):

  ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
  PCI: Discovered peer bus fe
  pci_bus 0000:fe: root bus resource [io  0x0000-0xffff]
  pci_bus 0000:fe: root bus resource [mem 0x00000000-0xffffffffff]
  pci 0000:fe:03.0: [8086:2d98] type 00 class 0x060000
  PCI: Discovered peer bus ff
  pci_bus 0000:ff: root bus resource [io  0x0000-0xffff]
  pci_bus 0000:ff: root bus resource [mem 0x00000000-0xffffffffff]
  pci 0000:ff:03.0: [8086:2d98] type 00 class 0x060000
  EDAC MC1: Giving out device to module i7core_edac.c controller i7 core #1: DEV 0000:fe:03.0 (INTERRUPT)
  EDAC PCI0: Giving out device to module i7core_edac controller EDAC PCI controller: DEV 0000:fe:03.0 (POLLED)
  EDAC MC0: Giving out device to module i7core_edac.c controller i7 core #0: DEV 0000:ff:03.0 (INTERRUPT)
  EDAC PCI1: Giving out device to module i7core_edac controller EDAC PCI controller: DEV 0000:ff:03.0 (POLLED)

Some of the problems are:

  - Firmware may have omitted the host bridges to [bus fe] and
    [bus ff] from the ACPI namespace because *it* is using those
    management devices, so EDAC blindly using them is a potential
    conflict.

  - pcibios_fixup_peer_bridges() only scans domain 0, so if this
    system had multiple domains, EDAC would only work on things in
    domain 0, ignoring other domains.

  - The PCI core can't do bus number assignment correctly for devices
    behind bridge PCI0.  The firmware told us [bus 00-ff] was
    available, so the core may assign bus number fe to some deep
    switch hierarchy.  But bus fe conflicts with the devices on the
    "peer bus fe".  This part is a firmware bug: it should have told
    us that PCI0 leads to [bus 00-fd], not [bus 00-ff].

  - The PCI core can't do resource assignment correctly for devices on
    [bus fe] and [bus ff].  It has no information about what MMIO and
    I/O port are routed to those buses, so it assumes *all* memory and
    I/O ports are routed there, which is clearly incorrect.  This part
    is a Linux bug; we really shouldn't be poking around for buses
    that ACPI didn't tell us about.

Bjorn

[1] https://bugs.freedesktop.org/attachment.cgi?id=136529

Re: Peer bridge fixup issue under multiple pci domain

[Zihan Yang]

Bjorn Helgaas <helgaas@kernel.org> =E4=BA=8E2018=E5=B9=B48=E6=9C=8828=E6=97=
=A5=E5=91=A8=E4=BA=8C =E4=B8=8B=E5=8D=887:35=E5=86=99=E9=81=93=EF=BC=9A
>
> [+cc EDAC folks, LKML]
>
> On Sat, Aug 25, 2018 at 10:58:57PM +0800, Zihan Yang wrote:
> > Hi all,
> >
> > I'm trying to use multiple pci domain in qemu q35, but I find there
> > might be some issues in peer bridge fixup.
> >
> > In short, pcibios_fixup_peer_bridges function assumes only one pci
> > domain (0) by default. This is OK when as qemu by default uses only
> > one pci domain too. However, if I add another host bridge which is
> > put into pci domain 1 by using _SEG, and a pcie_pci_bridge is attached
> > to the bus 1 under this new pci domain 1 rather than domain 0, the
> > kernel will recognize the bus 01 differently.
> >
> > More specifically, pcibios_fixup_peer_bridges only reads all the buses
> > under domain 0 but it can read the pci bus 01 in pci domain 1 and treat
> > it as a peer bus of 0000:00. The consequence is this 01 bus is recogniz=
ed
> > as 0000:01, but it should have been recognized as 0001:01.
> >
> > The host bus 0001:00 can be recognized so I guess pcibios_fixup_peer_br=
idges
> > needs updating to take care of multiple domains? Or is it just an bios =
issue?
> > I'm not quite sure and I'm open to any suggestions.
>
> Is there something that actually does not work, or is this just a
> concern that the code looks wrong?

Sorry for the late reply, currently it is just a concern because the
qemu part is
still ongoing and I'm not quite sure about the root cause. But my disussion=
 with
qemu developers indicate my issue might originates from incorrect AML, whic=
h
includes _SEG, _BBN and _CRS as you state below. I will try to locate the r=
eal
cause recently.

> pcibios_fixup_peer_bridges() is ancient history from before x86 used
> the ACPI namespace to discover host bridges.  It blindly probes for
> devices on buses 0-255, but as you say, only in domain 0.
>
> Using multiple PCI domains really requires ACPI support so we know
> what the other domains are (_SEG) and how to access their config space
> (MCFG).  When we do have ACPI support in the platform and the kernel,
> drivers/acpi/pci_root.c discovers all the host bridges in all domains
> via PNP0A03 or PNP0A08 devices in the ACPI namespace, and in most
> cases pcibios_fixup_peer_bridges() will do nothing.
>
> However, there *are* systems where the firmware does not expose all
> host bridges and in those cases, pcibios_fixup_peer_bridges() can be a
> problem.  For example, Intel processors often have management devices
> on bus 7f or ff.  If the ACPI namespace doesn't have a host bridge to
> those buses, pci_root.c won't find them, but
> pcibios_fixup_peer_bridges() *will*.

Thanks for clarifying, does it only affect bus 7f/ff, or does it
affect other busses
as well? If only management devices are affected, then I think it is
not the cause
of my issue. But thanks a lot for your detailed reply.

> This leads to several problems.  Here's a dmesg sample from [1]
> (found by googling for 'dmesg log "PCI: discovered peer bus ff"'):
>
>   ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
>   PCI: Discovered peer bus fe
>   pci_bus 0000:fe: root bus resource [io  0x0000-0xffff]
>   pci_bus 0000:fe: root bus resource [mem 0x00000000-0xffffffffff]
>   pci 0000:fe:03.0: [8086:2d98] type 00 class 0x060000
>   PCI: Discovered peer bus ff
>   pci_bus 0000:ff: root bus resource [io  0x0000-0xffff]
>   pci_bus 0000:ff: root bus resource [mem 0x00000000-0xffffffffff]
>   pci 0000:ff:03.0: [8086:2d98] type 00 class 0x060000
>   EDAC MC1: Giving out device to module i7core_edac.c controller i7 core =
#1: DEV 0000:fe:03.0 (INTERRUPT)
>   EDAC PCI0: Giving out device to module i7core_edac controller EDAC PCI =
controller: DEV 0000:fe:03.0 (POLLED)
>   EDAC MC0: Giving out device to module i7core_edac.c controller i7 core =
#0: DEV 0000:ff:03.0 (INTERRUPT)
>   EDAC PCI1: Giving out device to module i7core_edac controller EDAC PCI =
controller: DEV 0000:ff:03.0 (POLLED)
>
> Some of the problems are:
>
>   - Firmware may have omitted the host bridges to [bus fe] and
>     [bus ff] from the ACPI namespace because *it* is using those
>     management devices, so EDAC blindly using them is a potential
>     conflict.
>
>   - pcibios_fixup_peer_bridges() only scans domain 0, so if this
>     system had multiple domains, EDAC would only work on things in
>     domain 0, ignoring other domains.
>
>   - The PCI core can't do bus number assignment correctly for devices
>     behind bridge PCI0.  The firmware told us [bus 00-ff] was
>     available, so the core may assign bus number fe to some deep
>     switch hierarchy.  But bus fe conflicts with the devices on the
>     "peer bus fe".  This part is a firmware bug: it should have told
>     us that PCI0 leads to [bus 00-fd], not [bus 00-ff].
>
>   - The PCI core can't do resource assignment correctly for devices on
>     [bus fe] and [bus ff].  It has no information about what MMIO and
>     I/O port are routed to those buses, so it assumes *all* memory and
>     I/O ports are routed there, which is clearly incorrect.  This part
>     is a Linux bug; we really shouldn't be poking around for buses
>     that ACPI didn't tell us about.
>
> Bjorn
>
> [1] https://bugs.freedesktop.org/attachment.cgi?id=3D136529