* Fwd: PCIe root port programming
[not found] ` <CAH=XD58J+Yf-Rdz_bAJ-e9rU4ipu9opieFd0BuHeYAdX0nmrzg@mail.gmail.com>
@ 2016-08-12 21:14 ` Kallol Biswas
2016-08-15 17:24 ` Bjorn Helgaas
0 siblings, 1 reply; 8+ messages in thread
From: Kallol Biswas @ 2016-08-12 21:14 UTC (permalink / raw)
To: linux-pci, helgaas
I am trying to debug an issue that the PCIe memory
transactions are not routed through a host bridge.
The config transactions work fine. The primary, secondary and
subordinate bus numbers look fine for the PCIe tree under the host
bridge. The memory windows are programmed fine.
Can someone point me to the code that programs the chipset/root port
registers those direct processor transactions to a specific host
bridge?
Cat of /proc/mem is copied below.
/root/physmem -p c8100000 -s 4 -r
Operation:read to Addr: 0xc8100000 Value: 0
ffffffff ffffffff ffffffff ffffffff
The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
external switches/devices does not capture any mem transactions.
However, config transactions are visible in the trace.
root@xlink-target:~# cat /proc/iomem
00000000-00000fff : reserved
00001000-0009abff : System RAM
0009ac00-0009ffff : reserved
000c0000-000c7fff : Video ROM
000c8000-000c8fff : Adapter ROM
000e0000-000fffff : reserved
000f0000-000fffff : System ROM
00100000-78f36fff : System RAM
01000000-01828f14 : Kernel code
01828f15-01f3fb3f : Kernel data
020bb000-021fdfff : Kernel bss
30000000-37ffffff : Crash kernel
78f37000-79839fff : reserved
79116018-79116018 : APEI ERST
7911601c-79116021 : APEI ERST
79116028-79116039 : APEI ERST
79116040-7911604c : APEI ERST
79116050-7911804f : APEI ERST
7983a000-79d4afff : ACPI Non-volatile Storage
79d4b000-8fffffff : reserved
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
c6000000-c70fffff : PCI Bus 0000:2f
c6000000-c70fffff : PCI Bus 0000:30
c6000000-c6ffffff : 0000:30:00.0
c7000000-c701ffff : 0000:30:00.0
c7100000-c71fffff : PCI Bus 0000:31
c7100000-c711ffff : 0000:31:00.1
c7100000-c711ffff : igb
c7120000-c713ffff : 0000:31:00.0
c7120000-c713ffff : igb
c7140000-c7143fff : 0000:31:00.1
c7140000-c7143fff : igb
c7144000-c7147fff : 0000:31:00.0
c7144000-c7147fff : igb
c7300000-c730ffff : 0000:00:14.0
c7300000-c730ffff : xhci-hcd
c7310000-c73107ff : 0000:00:1f.2
c7310000-c73107ff : ahci
c7311000-c73113ff : 0000:00:1d.0
c7311000-c73113ff : ehci_hcd
c7312000-c73123ff : 0000:00:1a.0
c7312000-c73123ff : ehci_hcd
c7314000-c731400f : 0000:00:16.1
c7315000-c731500f : 0000:00:16.0
c7316000-c73167ff : 0000:00:11.4
c7316000-c73167ff : ahci
c7317000-c7317fff : 0000:00:05.4
c7ffc000-c7ffcfff : dmar1
c8000000-c8000000 : <BAD>
c8000001-c8000001 : <BAD>
c8000002-c8000002 : <BAD>
c8000003-c8000003 : <BAD>
c8100000-c91fffff : PCI Bus 0000:02
c8100000-c90fffff : PCI Bus 0000:03
c8100000-c90fffff : PCI Bus 0000:04
c8100000-c82fffff : PCI Bus 0000:05
c8100000-c81fffff : PCI Bus 0000:06
c8100000-c8100fff : 0000:06:00.0
c8200000-c82fffff : PCI Bus 0000:0a
c8200000-c8203fff : 0000:0a:00.0
c9100000-c913ffff : 0000:02:00.0
fb000000-fb8fffff : PCI Bus 0000:81
fb000000-fb8fffff : PCI Bus 0000:82
fb000000-fb7fffff : PCI Bus 0000:8c
fb000000-fb7fffff : 0000:8c:00.0
fb000000-fb7fffff : Plx_Mgr
fb800000-fb8fffff : PCI Bus 0000:87
fb800000-fb800fff : 0000:87:00.0
fb900000-fb900fff : 0000:80:05.4
fbffc000-fbffcfff : dmar0
fec00000-fecfffff : PNP0003:00
fec00000-fec003ff : IOAPIC 0
fec01000-fec013ff : IOAPIC 1
fec40000-fec403ff : IOAPIC 2
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed12000-fed1200f : pnp 00:01
fed12010-fed1201f : pnp 00:01
fed1b000-fed1bfff : pnp 00:01
fed1c000-fed44fff : reserved
fed1c000-fed3ffff : pnp 00:01
fed1f410-fed1f414 : iTCO_wdt.0.auto
fed45000-fed8bfff : pnp 00:01
fee00000-feefffff : pnp 00:01
fee00000-fee00fff : Local APIC
ff000000-ffffffff : reserved
ff000000-ffffffff : pnp 00:01
100000000-47fffffff : System RAM
480000000-4807fffff : PCI Bus 0000:02
480000000-4807fffff : PCI Bus 0000:03
480000000-4807fffff : PCI Bus 0000:04
480000000-4800fffff : PCI Bus 0000:05
480000000-4800fffff : PCI Bus 0000:06
480000000-480003fff : 0000:06:00.0
383fffe00000-383fffefffff : PCI Bus 0000:31
383fffe00000-383fffe1ffff : 0000:31:00.1
383fffe20000-383fffe3ffff : 0000:31:00.1
383fffe40000-383fffe5ffff : 0000:31:00.0
383fffe60000-383fffe7ffff : 0000:31:00.0
383ffff00000-383ffff03fff : 0000:00:04.7
383ffff00000-383ffff03fff : ioatdma
383ffff04000-383ffff07fff : 0000:00:04.6
383ffff04000-383ffff07fff : ioatdma
383ffff08000-383ffff0bfff : 0000:00:04.5
383ffff08000-383ffff0bfff : ioatdma
383ffff0c000-383ffff0ffff : 0000:00:04.4
383ffff0c000-383ffff0ffff : ioatdma
383ffff10000-383ffff13fff : 0000:00:04.3
383ffff10000-383ffff13fff : ioatdma
383ffff14000-383ffff17fff : 0000:00:04.2
383ffff14000-383ffff17fff : ioatdma
383ffff18000-383ffff1bfff : 0000:00:04.1
383ffff18000-383ffff1bfff : ioatdma
383ffff1c000-383ffff1ffff : 0000:00:04.0
383ffff1c000-383ffff1ffff : ioatdma
383ffff21000-383ffff210ff : 0000:00:1f.3
387fffe00000-387fffefffff : PCI Bus 0000:81
387fffe00000-387fffefffff : PCI Bus 0000:82
387fffe00000-387fffefffff : PCI Bus 0000:87
387fffe00000-387fffe03fff : 0000:87:00.0
387ffff00000-387ffff03fff : 0000:80:04.7
387ffff00000-387ffff03fff : ioatdma
387ffff04000-387ffff07fff : 0000:80:04.6
387ffff04000-387ffff07fff : ioatdma
387ffff08000-387ffff0bfff : 0000:80:04.5
387ffff08000-387ffff0bfff : ioatdma
387ffff0c000-387ffff0ffff : 0000:80:04.4
387ffff0c000-387ffff0ffff : ioatdma
387ffff10000-387ffff13fff : 0000:80:04.3
387ffff10000-387ffff13fff : ioatdma
387ffff14000-387ffff17fff : 0000:80:04.2
387ffff14000-387ffff17fff : ioatdma
387ffff18000-387ffff1bfff : 0000:80:04.1
387ffff18000-387ffff1bfff : ioatdma
387ffff1c000-387ffff1ffff : 0000:80:04.0
387ffff1c000-387ffff1ffff : ioatdma
Memory windows:
| \-1f.2
\-[0000:00]-+-00.0
+-01.0-[01]--
+-03.0-[02-2c]----00.0-[03-2c]----08.0-[04-2c]----00.0-[05-0a]--+-00.0-[06]----00.0
|
+-01.0-[07]--
|
+-02.0-[08]--
|
+-03.0-[09]--
|
\-1e.0-[0a]----00.0
+-03.2-[2d]--
root@xlink-target:~# lspci -v -s 0:03.0
00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3
v4/Xeon D PCI Express Root Port 3 (rev 01) (prog-if 00 [Normal
decode])
Flags: bus master, fast devsel, latency 0, IRQ 27
Bus: primary=00, secondary=02, subordinate=2c, sec-latency=0
Memory behind bridge: c8100000-c91fffff
Prefetchable memory behind bridge: 0000000480000000-00000004807fffff
root@xlink-target:~# lspci -v -s 02:00.0
02:00.0 PCI bridge: PLX Technology, Inc. PEX 8732 32-lane, 8-Port PCI
Express Gen 3 (8.0 GT/s) Switch (rev ca) (prog-if 00 [Normal decode])
Physical Slot: 4
Flags: bus master, fast devsel, latency 0, IRQ 32
Memory at c9100000 (32-bit, non-prefetchable) [size=256K]
Bus: primary=02, secondary=03, subordinate=2c, sec-latency=0
Memory behind bridge: c8100000-c90fffff
Prefetchable memory behind bridge: 0000000480000000-00000004807fffff
root@xlink-target:~# lspci -v -s 03:08.0
03:08.0 PCI bridge: PLX Technology, Inc. PEX 8732 32-lane, 8-Port PCI
Express Gen 3 (8.0 GT/s) Switch (rev ca) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 33
Bus: primary=03, secondary=04, subordinate=2c, sec-latency=0
Memory behind bridge: c8100000-c90fffff
Prefetchable memory behind bridge: 0000000480000000-00000004807fffff
root@xlink-target:~# lspci -v -s 04:00.0
04:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa) (prog-if
00 [Normal decode])
Physical Slot: 8-1
Flags: bus master, fast devsel, latency 0
Bus: primary=04, secondary=05, subordinate=0a, sec-latency=0
Memory behind bridge: c8100000-c82fffff
Prefetchable memory behind bridge: 0000000480000000-00000004800fffff
Capabilities: [40] Power Management version 3
root@xlink-target:~# lspci -v -s 05:00.0
05:00.0 PCI bridge: PLX Technology, Inc. Device 9797 (rev aa) (prog-if
00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 96
Bus: primary=05, secondary=06, subordinate=06, sec-latency=0
Memory behind bridge: c8100000-c81fffff
Prefetchable memory behind bridge: 0000000480000000-00000004800fffff
root@xlink-target:~# lspci -v -s 06:00.0
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Subsystem: TP-LINK Technologies Co., Ltd. TG-3468 Gigabit PCI Express
Network Adapter
Physical Slot: 1
Flags: bus master, fast devsel, latency 0
Memory at c8100000 (64-bit, non-prefetchable) [size=4K]
Memory at 480000000 (64-bit, prefetchable) [size=16K]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming
2016-08-12 21:14 ` Fwd: PCIe root port programming Kallol Biswas
@ 2016-08-15 17:24 ` Bjorn Helgaas
2016-08-15 18:39 ` Kallol Biswas
0 siblings, 1 reply; 8+ messages in thread
From: Bjorn Helgaas @ 2016-08-15 17:24 UTC (permalink / raw)
To: Kallol Biswas; +Cc: linux-pci
Hi Kallol,
Sorry I didn't respond to this earlier; I was on vacation for most of
last week.
On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
> I am trying to debug an issue that the PCIe memory
> transactions are not routed through a host bridge.
>
> The config transactions work fine. The primary, secondary and
> subordinate bus numbers look fine for the PCIe tree under the host
> bridge. The memory windows are programmed fine.
>
> Can someone point me to the code that programs the chipset/root port
> registers those direct processor transactions to a specific host
> bridge?
It looks like you have an x86 system with ACPI. In that case, we use
the ACPI pci_root.c host bridge driver. The chipset configuration for
routing processor memory transactions to host bridges is done by the
firmware, and the kernel doesn't touch it. There should be
information in the chipset spec about how to do this, but there's no
code in Linux itself that does it.
> Cat of /proc/mem is copied below.
>
> /root/physmem -p c8100000 -s 4 -r
> Operation:read to Addr: 0xc8100000 Value: 0
> ffffffff ffffffff ffffffff ffffffff
>
> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
> external switches/devices does not capture any mem transactions.
> However, config transactions are visible in the trace.
00:03.0 is not a host bridge; it's a Root Port, which logically
connects two PCI buses. A host bridge logically connects a processor
bus to a PCI bus.
> root@xlink-target:~# cat /proc/iomem
> 00000000-00000fff : reserved
> 00001000-0009abff : System RAM
> 0009ac00-0009ffff : reserved
> 000c0000-000c7fff : Video ROM
> 000c8000-000c8fff : Adapter ROM
> 000e0000-000fffff : reserved
> 000f0000-000fffff : System ROM
> 00100000-78f36fff : System RAM
> 01000000-01828f14 : Kernel code
> 01828f15-01f3fb3f : Kernel data
> 020bb000-021fdfff : Kernel bss
> 30000000-37ffffff : Crash kernel
> 78f37000-79839fff : reserved
> 79116018-79116018 : APEI ERST
> 7911601c-79116021 : APEI ERST
> 79116028-79116039 : APEI ERST
> 79116040-7911604c : APEI ERST
> 79116050-7911804f : APEI ERST
> 7983a000-79d4afff : ACPI Non-volatile Storage
> 79d4b000-8fffffff : reserved
> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> c6000000-c70fffff : PCI Bus 0000:2f
> c6000000-c70fffff : PCI Bus 0000:30
> c6000000-c6ffffff : 0000:30:00.0
> c7000000-c701ffff : 0000:30:00.0
> c7100000-c71fffff : PCI Bus 0000:31
> c7100000-c711ffff : 0000:31:00.1
> c7100000-c711ffff : igb
> c7120000-c713ffff : 0000:31:00.0
> c7120000-c713ffff : igb
> c7140000-c7143fff : 0000:31:00.1
> c7140000-c7143fff : igb
> c7144000-c7147fff : 0000:31:00.0
> c7144000-c7147fff : igb
> c7300000-c730ffff : 0000:00:14.0
> c7300000-c730ffff : xhci-hcd
> c7310000-c73107ff : 0000:00:1f.2
> c7310000-c73107ff : ahci
> c7311000-c73113ff : 0000:00:1d.0
> c7311000-c73113ff : ehci_hcd
> c7312000-c73123ff : 0000:00:1a.0
> c7312000-c73123ff : ehci_hcd
> c7314000-c731400f : 0000:00:16.1
> c7315000-c731500f : 0000:00:16.0
> c7316000-c73167ff : 0000:00:11.4
> c7316000-c73167ff : ahci
> c7317000-c7317fff : 0000:00:05.4
> c7ffc000-c7ffcfff : dmar1
> c8000000-c8000000 : <BAD>
> c8000001-c8000001 : <BAD>
> c8000002-c8000002 : <BAD>
> c8000003-c8000003 : <BAD>
Interesting, what are these from?
> c8100000-c91fffff : PCI Bus 0000:02
> c8100000-c90fffff : PCI Bus 0000:03
> c8100000-c90fffff : PCI Bus 0000:04
> c8100000-c82fffff : PCI Bus 0000:05
> c8100000-c81fffff : PCI Bus 0000:06
> c8100000-c8100fff : 0000:06:00.0
You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
above look fine, but the top-level host bridge window is missing.
The c8100000-c91fffff window is a window through the 00:03.0 Root
Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
memory transactions in the c8100000-c91fffff range from its primary
bus (bus 00) to its secondary bus (bus 02).
But we don't know how those transactions get to bus 00 in the first
place. The host bridge is the connection between the processor and
bus 00, and it should have a window that encloses c8100000-c91fffff.
Your dmesg log should have information about this bridge, e.g.,
something like this:
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
and there should be a line in /proc/iomem something like this:
c8100000-c9ffffff : PCI Bus 0000:00
Bjorn
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming
2016-08-15 17:24 ` Bjorn Helgaas
@ 2016-08-15 18:39 ` Kallol Biswas
2016-08-15 19:06 ` Bjorn Helgaas
0 siblings, 1 reply; 8+ messages in thread
From: Kallol Biswas @ 2016-08-15 18:39 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: linux-pci
[-- Attachment #1: Type: text/plain, Size: 7162 bytes --]
Thank you for the explanation.
Here is the relevant dmesg content:
[ 0.790457] PCI: root bus ff: using default resources
[ 0.790491] PCI host bridge to bus 0000:ff
[ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
[ 0.790495] pci_bus 0000:ff: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
0.795379] PCI host bridge to bus 0000:7f
[ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
[ 0.795383] pci_bus 0000:7f: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
0.809149] PCI host bridge to bus 0000:00
[ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.809153] pci_bus 0000:00: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
[ 0.852071] PCI host bridge to bus 0000:80
[ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
[ 0.852075] pci_bus 0000:80: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
Following lines are from output of /proc/iomem
c8000000-c8000000 : <BAD>
c8000001-c8000001 : <BAD>
c8000002-c8000002 : <BAD>
c8000003-c8000003 : <BAD>
c8100000-c91fffff : PCI Bus 0000:02
c8100000-c90fffff : PCI Bus 0000:03
c8100000-c90fffff : PCI Bus 0000:04
c8100000-c82fffff : PCI Bus 0000:05
c8100000-c81fffff : PCI Bus 0000:06
c8100000-c8100fff : 0000:06:00.0
c8200000-c82fffff : PCI Bus 0000:0a
c8200000-c8203fff : 0000:0a:00.0
c9100000-c913ffff : 0000:02:00.0
fb000000-fb8fffff : PCI Bus 0000:81
fb000000-fb8fffff : PCI Bus 0000:82
fb000000-fb7fffff : PCI Bus 0000:8c
fb000000-fb7fffff : 0000:8c:00.0
fb000000-fb7fffff : Plx_Mgr
fb800000-fb8fffff : PCI Bus 0000:87
fb800000-fb800fff : 0000:87:00.0
fb900000-fb900fff : 0000:80:05.4
I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
device is available after reboot, and initial PCIe bus enumeration
time.
The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
operation is done.
Attached is the complete /proc/iomem and do not find the line
c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
PCI Bus 0000:80? BAR access works fine through this bus.
On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> Hi Kallol,
>
> Sorry I didn't respond to this earlier; I was on vacation for most of
> last week.
>
> On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
>> I am trying to debug an issue that the PCIe memory
>> transactions are not routed through a host bridge.
>>
>> The config transactions work fine. The primary, secondary and
>> subordinate bus numbers look fine for the PCIe tree under the host
>> bridge. The memory windows are programmed fine.
>>
>> Can someone point me to the code that programs the chipset/root port
>> registers those direct processor transactions to a specific host
>> bridge?
>
> It looks like you have an x86 system with ACPI. In that case, we use
> the ACPI pci_root.c host bridge driver. The chipset configuration for
> routing processor memory transactions to host bridges is done by the
> firmware, and the kernel doesn't touch it. There should be
> information in the chipset spec about how to do this, but there's no
> code in Linux itself that does it.
>
>> Cat of /proc/mem is copied below.
>>
>> /root/physmem -p c8100000 -s 4 -r
>> Operation:read to Addr: 0xc8100000 Value: 0
>> ffffffff ffffffff ffffffff ffffffff
>>
>> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
>> external switches/devices does not capture any mem transactions.
>> However, config transactions are visible in the trace.
>
> 00:03.0 is not a host bridge; it's a Root Port, which logically
> connects two PCI buses. A host bridge logically connects a processor
> bus to a PCI bus.
>
>> root@xlink-target:~# cat /proc/iomem
>> 00000000-00000fff : reserved
>> 00001000-0009abff : System RAM
>> 0009ac00-0009ffff : reserved
>> 000c0000-000c7fff : Video ROM
>> 000c8000-000c8fff : Adapter ROM
>> 000e0000-000fffff : reserved
>> 000f0000-000fffff : System ROM
>> 00100000-78f36fff : System RAM
>> 01000000-01828f14 : Kernel code
>> 01828f15-01f3fb3f : Kernel data
>> 020bb000-021fdfff : Kernel bss
>> 30000000-37ffffff : Crash kernel
>> 78f37000-79839fff : reserved
>> 79116018-79116018 : APEI ERST
>> 7911601c-79116021 : APEI ERST
>> 79116028-79116039 : APEI ERST
>> 79116040-7911604c : APEI ERST
>> 79116050-7911804f : APEI ERST
>> 7983a000-79d4afff : ACPI Non-volatile Storage
>> 79d4b000-8fffffff : reserved
>> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> c6000000-c70fffff : PCI Bus 0000:2f
>> c6000000-c70fffff : PCI Bus 0000:30
>> c6000000-c6ffffff : 0000:30:00.0
>> c7000000-c701ffff : 0000:30:00.0
>> c7100000-c71fffff : PCI Bus 0000:31
>> c7100000-c711ffff : 0000:31:00.1
>> c7100000-c711ffff : igb
>> c7120000-c713ffff : 0000:31:00.0
>> c7120000-c713ffff : igb
>> c7140000-c7143fff : 0000:31:00.1
>> c7140000-c7143fff : igb
>> c7144000-c7147fff : 0000:31:00.0
>> c7144000-c7147fff : igb
>> c7300000-c730ffff : 0000:00:14.0
>> c7300000-c730ffff : xhci-hcd
>> c7310000-c73107ff : 0000:00:1f.2
>> c7310000-c73107ff : ahci
>> c7311000-c73113ff : 0000:00:1d.0
>> c7311000-c73113ff : ehci_hcd
>> c7312000-c73123ff : 0000:00:1a.0
>> c7312000-c73123ff : ehci_hcd
>> c7314000-c731400f : 0000:00:16.1
>> c7315000-c731500f : 0000:00:16.0
>> c7316000-c73167ff : 0000:00:11.4
>> c7316000-c73167ff : ahci
>> c7317000-c7317fff : 0000:00:05.4
>> c7ffc000-c7ffcfff : dmar1
>> c8000000-c8000000 : <BAD>
>> c8000001-c8000001 : <BAD>
>> c8000002-c8000002 : <BAD>
>> c8000003-c8000003 : <BAD>
>
> Interesting, what are these from?
>
>> c8100000-c91fffff : PCI Bus 0000:02
>> c8100000-c90fffff : PCI Bus 0000:03
>> c8100000-c90fffff : PCI Bus 0000:04
>> c8100000-c82fffff : PCI Bus 0000:05
>> c8100000-c81fffff : PCI Bus 0000:06
>> c8100000-c8100fff : 0000:06:00.0
>
> You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
> above look fine, but the top-level host bridge window is missing.
>
> The c8100000-c91fffff window is a window through the 00:03.0 Root
> Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
> memory transactions in the c8100000-c91fffff range from its primary
> bus (bus 00) to its secondary bus (bus 02).
>
> But we don't know how those transactions get to bus 00 in the first
> place. The host bridge is the connection between the processor and
> bus 00, and it should have a window that encloses c8100000-c91fffff.
>
> Your dmesg log should have information about this bridge, e.g.,
> something like this:
>
> ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
> PCI host bridge to bus 0000:00
> pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
>
> and there should be a line in /proc/iomem something like this:
>
> c8100000-c9ffffff : PCI Bus 0000:00
>
> Bjorn
[-- Attachment #2: iomem.txt --]
[-- Type: text/plain, Size: 4888 bytes --]
00000000-00000fff : reserved
00001000-0009abff : System RAM
0009ac00-0009ffff : reserved
000c0000-000c7fff : Video ROM
000c8000-000c8fff : Adapter ROM
000e0000-000fffff : reserved
000f0000-000fffff : System ROM
00100000-78f36fff : System RAM
01000000-01828f14 : Kernel code
01828f15-01f3fb3f : Kernel data
020bb000-021fdfff : Kernel bss
30000000-37ffffff : Crash kernel
78f37000-79839fff : reserved
79116018-79116018 : APEI ERST
7911601c-79116021 : APEI ERST
79116028-79116039 : APEI ERST
79116040-7911604c : APEI ERST
79116050-7911804f : APEI ERST
7983a000-79d4afff : ACPI Non-volatile Storage
79d4b000-8fffffff : reserved
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
c6000000-c70fffff : PCI Bus 0000:2f
c6000000-c70fffff : PCI Bus 0000:30
c6000000-c6ffffff : 0000:30:00.0
c7000000-c701ffff : 0000:30:00.0
c7100000-c71fffff : PCI Bus 0000:31
c7100000-c711ffff : 0000:31:00.1
c7100000-c711ffff : igb
c7120000-c713ffff : 0000:31:00.0
c7120000-c713ffff : igb
c7140000-c7143fff : 0000:31:00.1
c7140000-c7143fff : igb
c7144000-c7147fff : 0000:31:00.0
c7144000-c7147fff : igb
c7300000-c730ffff : 0000:00:14.0
c7300000-c730ffff : xhci-hcd
c7310000-c73107ff : 0000:00:1f.2
c7310000-c73107ff : ahci
c7311000-c73113ff : 0000:00:1d.0
c7311000-c73113ff : ehci_hcd
c7312000-c73123ff : 0000:00:1a.0
c7312000-c73123ff : ehci_hcd
c7314000-c731400f : 0000:00:16.1
c7315000-c731500f : 0000:00:16.0
c7316000-c73167ff : 0000:00:11.4
c7316000-c73167ff : ahci
c7317000-c7317fff : 0000:00:05.4
c7ffc000-c7ffcfff : dmar1
c8000000-c8000000 : <BAD>
c8000001-c8000001 : <BAD>
c8000002-c8000002 : <BAD>
c8000003-c8000003 : <BAD>
c8100000-c91fffff : PCI Bus 0000:02
c8100000-c90fffff : PCI Bus 0000:03
c8100000-c90fffff : PCI Bus 0000:04
c8100000-c82fffff : PCI Bus 0000:05
c8100000-c81fffff : PCI Bus 0000:06
c8100000-c8100fff : 0000:06:00.0
c8200000-c82fffff : PCI Bus 0000:0a
c8200000-c8203fff : 0000:0a:00.0
c9100000-c913ffff : 0000:02:00.0
fb000000-fb8fffff : PCI Bus 0000:81
fb000000-fb8fffff : PCI Bus 0000:82
fb000000-fb7fffff : PCI Bus 0000:8c
fb000000-fb7fffff : 0000:8c:00.0
fb000000-fb7fffff : Plx_Mgr
fb800000-fb8fffff : PCI Bus 0000:87
fb800000-fb800fff : 0000:87:00.0
fb900000-fb900fff : 0000:80:05.4
fbffc000-fbffcfff : dmar0
fec00000-fecfffff : PNP0003:00
fec00000-fec003ff : IOAPIC 0
fec01000-fec013ff : IOAPIC 1
fec40000-fec403ff : IOAPIC 2
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed12000-fed1200f : pnp 00:01
fed12010-fed1201f : pnp 00:01
fed1b000-fed1bfff : pnp 00:01
fed1c000-fed44fff : reserved
fed1c000-fed3ffff : pnp 00:01
fed1f410-fed1f414 : iTCO_wdt.0.auto
fed45000-fed8bfff : pnp 00:01
fee00000-feefffff : pnp 00:01
fee00000-fee00fff : Local APIC
ff000000-ffffffff : reserved
ff000000-ffffffff : pnp 00:01
100000000-47fffffff : System RAM
480000000-4807fffff : PCI Bus 0000:02
480000000-4807fffff : PCI Bus 0000:03
480000000-4807fffff : PCI Bus 0000:04
480000000-4800fffff : PCI Bus 0000:05
480000000-4800fffff : PCI Bus 0000:06
480000000-480003fff : 0000:06:00.0
383fffe00000-383fffefffff : PCI Bus 0000:31
383fffe00000-383fffe1ffff : 0000:31:00.1
383fffe20000-383fffe3ffff : 0000:31:00.1
383fffe40000-383fffe5ffff : 0000:31:00.0
383fffe60000-383fffe7ffff : 0000:31:00.0
383ffff00000-383ffff03fff : 0000:00:04.7
383ffff00000-383ffff03fff : ioatdma
383ffff04000-383ffff07fff : 0000:00:04.6
383ffff04000-383ffff07fff : ioatdma
383ffff08000-383ffff0bfff : 0000:00:04.5
383ffff08000-383ffff0bfff : ioatdma
383ffff0c000-383ffff0ffff : 0000:00:04.4
383ffff0c000-383ffff0ffff : ioatdma
383ffff10000-383ffff13fff : 0000:00:04.3
383ffff10000-383ffff13fff : ioatdma
383ffff14000-383ffff17fff : 0000:00:04.2
383ffff14000-383ffff17fff : ioatdma
383ffff18000-383ffff1bfff : 0000:00:04.1
383ffff18000-383ffff1bfff : ioatdma
383ffff1c000-383ffff1ffff : 0000:00:04.0
383ffff1c000-383ffff1ffff : ioatdma
383ffff21000-383ffff210ff : 0000:00:1f.3
387fffe00000-387fffefffff : PCI Bus 0000:81
387fffe00000-387fffefffff : PCI Bus 0000:82
387fffe00000-387fffefffff : PCI Bus 0000:87
387fffe00000-387fffe03fff : 0000:87:00.0
387ffff00000-387ffff03fff : 0000:80:04.7
387ffff00000-387ffff03fff : ioatdma
387ffff04000-387ffff07fff : 0000:80:04.6
387ffff04000-387ffff07fff : ioatdma
387ffff08000-387ffff0bfff : 0000:80:04.5
387ffff08000-387ffff0bfff : ioatdma
387ffff0c000-387ffff0ffff : 0000:80:04.4
387ffff0c000-387ffff0ffff : ioatdma
387ffff10000-387ffff13fff : 0000:80:04.3
387ffff10000-387ffff13fff : ioatdma
387ffff14000-387ffff17fff : 0000:80:04.2
387ffff14000-387ffff17fff : ioatdma
387ffff18000-387ffff1bfff : 0000:80:04.1
387ffff18000-387ffff1bfff : ioatdma
387ffff1c000-387ffff1ffff : 0000:80:04.0
387ffff1c000-387ffff1ffff : ioatdma
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming
2016-08-15 18:39 ` Kallol Biswas
@ 2016-08-15 19:06 ` Bjorn Helgaas
2016-08-15 20:43 ` Kallol Biswas
0 siblings, 1 reply; 8+ messages in thread
From: Bjorn Helgaas @ 2016-08-15 19:06 UTC (permalink / raw)
To: Kallol Biswas; +Cc: linux-pci
On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote:
> Thank you for the explanation.
>
>
> Here is the relevant dmesg content:
>
> [ 0.790457] PCI: root bus ff: using default resources
> [ 0.790491] PCI host bridge to bus 0000:ff
> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
> [ 0.790495] pci_bus 0000:ff: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
>
> 0.795379] PCI host bridge to bus 0000:7f
> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
> [ 0.795383] pci_bus 0000:7f: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
>
> 0.809149] PCI host bridge to bus 0000:00
> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
> [ 0.809153] pci_bus 0000:00: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
This is the host bridge that claims to lead to bus 0000:00.
But the I/O and memory resources of all these bridges are wrong. The
ranges they show are CPU addresses, not PCI bus addresses, so they
cannot overlap.
It could be that you're using a non-standard host bridge driver that
did not apply the translation between CPU addresses and PCI bus
addresses. In ACPI, this is the _TRA property.
> [ 0.852071] PCI host bridge to bus 0000:80
> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
> [ 0.852075] pci_bus 0000:80: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
>
> Following lines are from output of /proc/iomem
>
> c8000000-c8000000 : <BAD>
> c8000001-c8000001 : <BAD>
> c8000002-c8000002 : <BAD>
> c8000003-c8000003 : <BAD>
> c8100000-c91fffff : PCI Bus 0000:02
> c8100000-c90fffff : PCI Bus 0000:03
> c8100000-c90fffff : PCI Bus 0000:04
> c8100000-c82fffff : PCI Bus 0000:05
> c8100000-c81fffff : PCI Bus 0000:06
> c8100000-c8100fff : 0000:06:00.0
> c8200000-c82fffff : PCI Bus 0000:0a
> c8200000-c8203fff : 0000:0a:00.0
> c9100000-c913ffff : 0000:02:00.0
> fb000000-fb8fffff : PCI Bus 0000:81
> fb000000-fb8fffff : PCI Bus 0000:82
> fb000000-fb7fffff : PCI Bus 0000:8c
> fb000000-fb7fffff : 0000:8c:00.0
> fb000000-fb7fffff : Plx_Mgr
> fb800000-fb8fffff : PCI Bus 0000:87
> fb800000-fb800fff : 0000:87:00.0
> fb900000-fb900fff : 0000:80:05.4
>
>
> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
> device is available after reboot, and initial PCIe bus enumeration
> time.
>
> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
> operation is done.
>
> Attached is the complete /proc/iomem and do not find the line
> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
> PCI Bus 0000:80? BAR access works fine through this bus.
>
>
>
> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> > Hi Kallol,
> >
> > Sorry I didn't respond to this earlier; I was on vacation for most of
> > last week.
> >
> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
> >> I am trying to debug an issue that the PCIe memory
> >> transactions are not routed through a host bridge.
> >>
> >> The config transactions work fine. The primary, secondary and
> >> subordinate bus numbers look fine for the PCIe tree under the host
> >> bridge. The memory windows are programmed fine.
> >>
> >> Can someone point me to the code that programs the chipset/root port
> >> registers those direct processor transactions to a specific host
> >> bridge?
> >
> > It looks like you have an x86 system with ACPI. In that case, we use
> > the ACPI pci_root.c host bridge driver. The chipset configuration for
> > routing processor memory transactions to host bridges is done by the
> > firmware, and the kernel doesn't touch it. There should be
> > information in the chipset spec about how to do this, but there's no
> > code in Linux itself that does it.
> >
> >> Cat of /proc/mem is copied below.
> >>
> >> /root/physmem -p c8100000 -s 4 -r
> >> Operation:read to Addr: 0xc8100000 Value: 0
> >> ffffffff ffffffff ffffffff ffffffff
> >>
> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
> >> external switches/devices does not capture any mem transactions.
> >> However, config transactions are visible in the trace.
> >
> > 00:03.0 is not a host bridge; it's a Root Port, which logically
> > connects two PCI buses. A host bridge logically connects a processor
> > bus to a PCI bus.
> >
> >> root@xlink-target:~# cat /proc/iomem
> >> 00000000-00000fff : reserved
> >> 00001000-0009abff : System RAM
> >> 0009ac00-0009ffff : reserved
> >> 000c0000-000c7fff : Video ROM
> >> 000c8000-000c8fff : Adapter ROM
> >> 000e0000-000fffff : reserved
> >> 000f0000-000fffff : System ROM
> >> 00100000-78f36fff : System RAM
> >> 01000000-01828f14 : Kernel code
> >> 01828f15-01f3fb3f : Kernel data
> >> 020bb000-021fdfff : Kernel bss
> >> 30000000-37ffffff : Crash kernel
> >> 78f37000-79839fff : reserved
> >> 79116018-79116018 : APEI ERST
> >> 7911601c-79116021 : APEI ERST
> >> 79116028-79116039 : APEI ERST
> >> 79116040-7911604c : APEI ERST
> >> 79116050-7911804f : APEI ERST
> >> 7983a000-79d4afff : ACPI Non-volatile Storage
> >> 79d4b000-8fffffff : reserved
> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> >> c6000000-c70fffff : PCI Bus 0000:2f
> >> c6000000-c70fffff : PCI Bus 0000:30
> >> c6000000-c6ffffff : 0000:30:00.0
> >> c7000000-c701ffff : 0000:30:00.0
> >> c7100000-c71fffff : PCI Bus 0000:31
> >> c7100000-c711ffff : 0000:31:00.1
> >> c7100000-c711ffff : igb
> >> c7120000-c713ffff : 0000:31:00.0
> >> c7120000-c713ffff : igb
> >> c7140000-c7143fff : 0000:31:00.1
> >> c7140000-c7143fff : igb
> >> c7144000-c7147fff : 0000:31:00.0
> >> c7144000-c7147fff : igb
> >> c7300000-c730ffff : 0000:00:14.0
> >> c7300000-c730ffff : xhci-hcd
> >> c7310000-c73107ff : 0000:00:1f.2
> >> c7310000-c73107ff : ahci
> >> c7311000-c73113ff : 0000:00:1d.0
> >> c7311000-c73113ff : ehci_hcd
> >> c7312000-c73123ff : 0000:00:1a.0
> >> c7312000-c73123ff : ehci_hcd
> >> c7314000-c731400f : 0000:00:16.1
> >> c7315000-c731500f : 0000:00:16.0
> >> c7316000-c73167ff : 0000:00:11.4
> >> c7316000-c73167ff : ahci
> >> c7317000-c7317fff : 0000:00:05.4
> >> c7ffc000-c7ffcfff : dmar1
> >> c8000000-c8000000 : <BAD>
> >> c8000001-c8000001 : <BAD>
> >> c8000002-c8000002 : <BAD>
> >> c8000003-c8000003 : <BAD>
> >
> > Interesting, what are these from?
> >
> >> c8100000-c91fffff : PCI Bus 0000:02
> >> c8100000-c90fffff : PCI Bus 0000:03
> >> c8100000-c90fffff : PCI Bus 0000:04
> >> c8100000-c82fffff : PCI Bus 0000:05
> >> c8100000-c81fffff : PCI Bus 0000:06
> >> c8100000-c8100fff : 0000:06:00.0
> >
> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
> > above look fine, but the top-level host bridge window is missing.
> >
> > The c8100000-c91fffff window is a window through the 00:03.0 Root
> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
> > memory transactions in the c8100000-c91fffff range from its primary
> > bus (bus 00) to its secondary bus (bus 02).
> >
> > But we don't know how those transactions get to bus 00 in the first
> > place. The host bridge is the connection between the processor and
> > bus 00, and it should have a window that encloses c8100000-c91fffff.
> >
> > Your dmesg log should have information about this bridge, e.g.,
> > something like this:
> >
> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
> > PCI host bridge to bus 0000:00
> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
> >
> > and there should be a line in /proc/iomem something like this:
> >
> > c8100000-c9ffffff : PCI Bus 0000:00
> >
> > Bjorn
> 00000000-00000fff : reserved
> 00001000-0009abff : System RAM
> 0009ac00-0009ffff : reserved
> 000c0000-000c7fff : Video ROM
> 000c8000-000c8fff : Adapter ROM
> 000e0000-000fffff : reserved
> 000f0000-000fffff : System ROM
> 00100000-78f36fff : System RAM
> 01000000-01828f14 : Kernel code
> 01828f15-01f3fb3f : Kernel data
> 020bb000-021fdfff : Kernel bss
> 30000000-37ffffff : Crash kernel
> 78f37000-79839fff : reserved
> 79116018-79116018 : APEI ERST
> 7911601c-79116021 : APEI ERST
> 79116028-79116039 : APEI ERST
> 79116040-7911604c : APEI ERST
> 79116050-7911804f : APEI ERST
> 7983a000-79d4afff : ACPI Non-volatile Storage
> 79d4b000-8fffffff : reserved
> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> c6000000-c70fffff : PCI Bus 0000:2f
> c6000000-c70fffff : PCI Bus 0000:30
> c6000000-c6ffffff : 0000:30:00.0
> c7000000-c701ffff : 0000:30:00.0
> c7100000-c71fffff : PCI Bus 0000:31
> c7100000-c711ffff : 0000:31:00.1
> c7100000-c711ffff : igb
> c7120000-c713ffff : 0000:31:00.0
> c7120000-c713ffff : igb
> c7140000-c7143fff : 0000:31:00.1
> c7140000-c7143fff : igb
> c7144000-c7147fff : 0000:31:00.0
> c7144000-c7147fff : igb
> c7300000-c730ffff : 0000:00:14.0
> c7300000-c730ffff : xhci-hcd
> c7310000-c73107ff : 0000:00:1f.2
> c7310000-c73107ff : ahci
> c7311000-c73113ff : 0000:00:1d.0
> c7311000-c73113ff : ehci_hcd
> c7312000-c73123ff : 0000:00:1a.0
> c7312000-c73123ff : ehci_hcd
> c7314000-c731400f : 0000:00:16.1
> c7315000-c731500f : 0000:00:16.0
> c7316000-c73167ff : 0000:00:11.4
> c7316000-c73167ff : ahci
> c7317000-c7317fff : 0000:00:05.4
> c7ffc000-c7ffcfff : dmar1
> c8000000-c8000000 : <BAD>
> c8000001-c8000001 : <BAD>
> c8000002-c8000002 : <BAD>
> c8000003-c8000003 : <BAD>
> c8100000-c91fffff : PCI Bus 0000:02
> c8100000-c90fffff : PCI Bus 0000:03
> c8100000-c90fffff : PCI Bus 0000:04
> c8100000-c82fffff : PCI Bus 0000:05
> c8100000-c81fffff : PCI Bus 0000:06
> c8100000-c8100fff : 0000:06:00.0
> c8200000-c82fffff : PCI Bus 0000:0a
> c8200000-c8203fff : 0000:0a:00.0
> c9100000-c913ffff : 0000:02:00.0
> fb000000-fb8fffff : PCI Bus 0000:81
> fb000000-fb8fffff : PCI Bus 0000:82
> fb000000-fb7fffff : PCI Bus 0000:8c
> fb000000-fb7fffff : 0000:8c:00.0
> fb000000-fb7fffff : Plx_Mgr
> fb800000-fb8fffff : PCI Bus 0000:87
> fb800000-fb800fff : 0000:87:00.0
> fb900000-fb900fff : 0000:80:05.4
> fbffc000-fbffcfff : dmar0
> fec00000-fecfffff : PNP0003:00
> fec00000-fec003ff : IOAPIC 0
> fec01000-fec013ff : IOAPIC 1
> fec40000-fec403ff : IOAPIC 2
> fed00000-fed003ff : HPET 0
> fed00000-fed003ff : PNP0103:00
> fed12000-fed1200f : pnp 00:01
> fed12010-fed1201f : pnp 00:01
> fed1b000-fed1bfff : pnp 00:01
> fed1c000-fed44fff : reserved
> fed1c000-fed3ffff : pnp 00:01
> fed1f410-fed1f414 : iTCO_wdt.0.auto
> fed45000-fed8bfff : pnp 00:01
> fee00000-feefffff : pnp 00:01
> fee00000-fee00fff : Local APIC
> ff000000-ffffffff : reserved
> ff000000-ffffffff : pnp 00:01
> 100000000-47fffffff : System RAM
> 480000000-4807fffff : PCI Bus 0000:02
> 480000000-4807fffff : PCI Bus 0000:03
> 480000000-4807fffff : PCI Bus 0000:04
> 480000000-4800fffff : PCI Bus 0000:05
> 480000000-4800fffff : PCI Bus 0000:06
> 480000000-480003fff : 0000:06:00.0
> 383fffe00000-383fffefffff : PCI Bus 0000:31
> 383fffe00000-383fffe1ffff : 0000:31:00.1
> 383fffe20000-383fffe3ffff : 0000:31:00.1
> 383fffe40000-383fffe5ffff : 0000:31:00.0
> 383fffe60000-383fffe7ffff : 0000:31:00.0
> 383ffff00000-383ffff03fff : 0000:00:04.7
> 383ffff00000-383ffff03fff : ioatdma
> 383ffff04000-383ffff07fff : 0000:00:04.6
> 383ffff04000-383ffff07fff : ioatdma
> 383ffff08000-383ffff0bfff : 0000:00:04.5
> 383ffff08000-383ffff0bfff : ioatdma
> 383ffff0c000-383ffff0ffff : 0000:00:04.4
> 383ffff0c000-383ffff0ffff : ioatdma
> 383ffff10000-383ffff13fff : 0000:00:04.3
> 383ffff10000-383ffff13fff : ioatdma
> 383ffff14000-383ffff17fff : 0000:00:04.2
> 383ffff14000-383ffff17fff : ioatdma
> 383ffff18000-383ffff1bfff : 0000:00:04.1
> 383ffff18000-383ffff1bfff : ioatdma
> 383ffff1c000-383ffff1ffff : 0000:00:04.0
> 383ffff1c000-383ffff1ffff : ioatdma
> 383ffff21000-383ffff210ff : 0000:00:1f.3
> 387fffe00000-387fffefffff : PCI Bus 0000:81
> 387fffe00000-387fffefffff : PCI Bus 0000:82
> 387fffe00000-387fffefffff : PCI Bus 0000:87
> 387fffe00000-387fffe03fff : 0000:87:00.0
> 387ffff00000-387ffff03fff : 0000:80:04.7
> 387ffff00000-387ffff03fff : ioatdma
> 387ffff04000-387ffff07fff : 0000:80:04.6
> 387ffff04000-387ffff07fff : ioatdma
> 387ffff08000-387ffff0bfff : 0000:80:04.5
> 387ffff08000-387ffff0bfff : ioatdma
> 387ffff0c000-387ffff0ffff : 0000:80:04.4
> 387ffff0c000-387ffff0ffff : ioatdma
> 387ffff10000-387ffff13fff : 0000:80:04.3
> 387ffff10000-387ffff13fff : ioatdma
> 387ffff14000-387ffff17fff : 0000:80:04.2
> 387ffff14000-387ffff17fff : ioatdma
> 387ffff18000-387ffff1bfff : 0000:80:04.1
> 387ffff18000-387ffff1bfff : ioatdma
> 387ffff1c000-387ffff1ffff : 0000:80:04.0
> 387ffff1c000-387ffff1ffff : ioatdma
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming
2016-08-15 19:06 ` Bjorn Helgaas
@ 2016-08-15 20:43 ` Kallol Biswas
2016-08-15 21:07 ` Bjorn Helgaas
0 siblings, 1 reply; 8+ messages in thread
From: Kallol Biswas @ 2016-08-15 20:43 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: linux-pci
The kernel source code is 4.4. I have not done any modification except
pre-allocated bus numbers and memory window for a DS port to allow
successful hot plug operation of a PCIe port.
0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored)
[ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7
window] (ignored)
[ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff
window] (ignored)
[ 0.611924] acpi PNP0A08:00: host bridge window [mem
0x000a0000-0x000bffff window] (ignored)
[ 0.611926] acpi PNP0A08:00: host bridge window [mem
0xfedb0000-0xfedb000f window] (ignored)
[ 0.611928] acpi PNP0A08:00: host bridge window [mem
0xfedc0000-0xfedc000f window] (ignored)
[ 0.611929] acpi PNP0A08:00: host bridge window [mem
0x90000000-0xc7ffbfff window] (ignored)
[ 0.611931] acpi PNP0A08:00: host bridge window [mem
0x380000000000-0x383fffffffff window] (ignored)
[ 0.611932] PCI: root bus 00: using default resources
[ 0.612217] PCI host bridge to bus 0000:00
[ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.612220] pci_bus 0000:00: root bus resource [mem
0x00000000-0x3fffffffffff]
[ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e]
kernel command line:
BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb
root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro
pci=realloc,nocrs,assign-busses pcie_ports=native
pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet
splash crashkernel=384M-:128M vt.handoff=7
On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote:
>> Thank you for the explanation.
>>
>>
>> Here is the relevant dmesg content:
>>
>> [ 0.790457] PCI: root bus ff: using default resources
>> [ 0.790491] PCI host bridge to bus 0000:ff
>> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
>> [ 0.790495] pci_bus 0000:ff: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
>>
>> 0.795379] PCI host bridge to bus 0000:7f
>> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
>> [ 0.795383] pci_bus 0000:7f: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
>>
>> 0.809149] PCI host bridge to bus 0000:00
>> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
>> [ 0.809153] pci_bus 0000:00: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
>
> This is the host bridge that claims to lead to bus 0000:00.
>
> But the I/O and memory resources of all these bridges are wrong. The
> ranges they show are CPU addresses, not PCI bus addresses, so they
> cannot overlap.
>
> It could be that you're using a non-standard host bridge driver that
> did not apply the translation between CPU addresses and PCI bus
> addresses. In ACPI, this is the _TRA property.
>
>> [ 0.852071] PCI host bridge to bus 0000:80
>> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
>> [ 0.852075] pci_bus 0000:80: root bus resource [mem
>> 0x00000000-0x3fffffffffff]
>> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
>>
>> Following lines are from output of /proc/iomem
>>
>> c8000000-c8000000 : <BAD>
>> c8000001-c8000001 : <BAD>
>> c8000002-c8000002 : <BAD>
>> c8000003-c8000003 : <BAD>
>> c8100000-c91fffff : PCI Bus 0000:02
>> c8100000-c90fffff : PCI Bus 0000:03
>> c8100000-c90fffff : PCI Bus 0000:04
>> c8100000-c82fffff : PCI Bus 0000:05
>> c8100000-c81fffff : PCI Bus 0000:06
>> c8100000-c8100fff : 0000:06:00.0
>> c8200000-c82fffff : PCI Bus 0000:0a
>> c8200000-c8203fff : 0000:0a:00.0
>> c9100000-c913ffff : 0000:02:00.0
>> fb000000-fb8fffff : PCI Bus 0000:81
>> fb000000-fb8fffff : PCI Bus 0000:82
>> fb000000-fb7fffff : PCI Bus 0000:8c
>> fb000000-fb7fffff : 0000:8c:00.0
>> fb000000-fb7fffff : Plx_Mgr
>> fb800000-fb8fffff : PCI Bus 0000:87
>> fb800000-fb800fff : 0000:87:00.0
>> fb900000-fb900fff : 0000:80:05.4
>>
>>
>> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
>> device is available after reboot, and initial PCIe bus enumeration
>> time.
>>
>> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
>> operation is done.
>>
>> Attached is the complete /proc/iomem and do not find the line
>> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
>> PCI Bus 0000:80? BAR access works fine through this bus.
>>
>>
>>
>> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>> > Hi Kallol,
>> >
>> > Sorry I didn't respond to this earlier; I was on vacation for most of
>> > last week.
>> >
>> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
>> >> I am trying to debug an issue that the PCIe memory
>> >> transactions are not routed through a host bridge.
>> >>
>> >> The config transactions work fine. The primary, secondary and
>> >> subordinate bus numbers look fine for the PCIe tree under the host
>> >> bridge. The memory windows are programmed fine.
>> >>
>> >> Can someone point me to the code that programs the chipset/root port
>> >> registers those direct processor transactions to a specific host
>> >> bridge?
>> >
>> > It looks like you have an x86 system with ACPI. In that case, we use
>> > the ACPI pci_root.c host bridge driver. The chipset configuration for
>> > routing processor memory transactions to host bridges is done by the
>> > firmware, and the kernel doesn't touch it. There should be
>> > information in the chipset spec about how to do this, but there's no
>> > code in Linux itself that does it.
>> >
>> >> Cat of /proc/mem is copied below.
>> >>
>> >> /root/physmem -p c8100000 -s 4 -r
>> >> Operation:read to Addr: 0xc8100000 Value: 0
>> >> ffffffff ffffffff ffffffff ffffffff
>> >>
>> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
>> >> external switches/devices does not capture any mem transactions.
>> >> However, config transactions are visible in the trace.
>> >
>> > 00:03.0 is not a host bridge; it's a Root Port, which logically
>> > connects two PCI buses. A host bridge logically connects a processor
>> > bus to a PCI bus.
>> >
>> >> root@xlink-target:~# cat /proc/iomem
>> >> 00000000-00000fff : reserved
>> >> 00001000-0009abff : System RAM
>> >> 0009ac00-0009ffff : reserved
>> >> 000c0000-000c7fff : Video ROM
>> >> 000c8000-000c8fff : Adapter ROM
>> >> 000e0000-000fffff : reserved
>> >> 000f0000-000fffff : System ROM
>> >> 00100000-78f36fff : System RAM
>> >> 01000000-01828f14 : Kernel code
>> >> 01828f15-01f3fb3f : Kernel data
>> >> 020bb000-021fdfff : Kernel bss
>> >> 30000000-37ffffff : Crash kernel
>> >> 78f37000-79839fff : reserved
>> >> 79116018-79116018 : APEI ERST
>> >> 7911601c-79116021 : APEI ERST
>> >> 79116028-79116039 : APEI ERST
>> >> 79116040-7911604c : APEI ERST
>> >> 79116050-7911804f : APEI ERST
>> >> 7983a000-79d4afff : ACPI Non-volatile Storage
>> >> 79d4b000-8fffffff : reserved
>> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> >> c6000000-c70fffff : PCI Bus 0000:2f
>> >> c6000000-c70fffff : PCI Bus 0000:30
>> >> c6000000-c6ffffff : 0000:30:00.0
>> >> c7000000-c701ffff : 0000:30:00.0
>> >> c7100000-c71fffff : PCI Bus 0000:31
>> >> c7100000-c711ffff : 0000:31:00.1
>> >> c7100000-c711ffff : igb
>> >> c7120000-c713ffff : 0000:31:00.0
>> >> c7120000-c713ffff : igb
>> >> c7140000-c7143fff : 0000:31:00.1
>> >> c7140000-c7143fff : igb
>> >> c7144000-c7147fff : 0000:31:00.0
>> >> c7144000-c7147fff : igb
>> >> c7300000-c730ffff : 0000:00:14.0
>> >> c7300000-c730ffff : xhci-hcd
>> >> c7310000-c73107ff : 0000:00:1f.2
>> >> c7310000-c73107ff : ahci
>> >> c7311000-c73113ff : 0000:00:1d.0
>> >> c7311000-c73113ff : ehci_hcd
>> >> c7312000-c73123ff : 0000:00:1a.0
>> >> c7312000-c73123ff : ehci_hcd
>> >> c7314000-c731400f : 0000:00:16.1
>> >> c7315000-c731500f : 0000:00:16.0
>> >> c7316000-c73167ff : 0000:00:11.4
>> >> c7316000-c73167ff : ahci
>> >> c7317000-c7317fff : 0000:00:05.4
>> >> c7ffc000-c7ffcfff : dmar1
>> >> c8000000-c8000000 : <BAD>
>> >> c8000001-c8000001 : <BAD>
>> >> c8000002-c8000002 : <BAD>
>> >> c8000003-c8000003 : <BAD>
>> >
>> > Interesting, what are these from?
>> >
>> >> c8100000-c91fffff : PCI Bus 0000:02
>> >> c8100000-c90fffff : PCI Bus 0000:03
>> >> c8100000-c90fffff : PCI Bus 0000:04
>> >> c8100000-c82fffff : PCI Bus 0000:05
>> >> c8100000-c81fffff : PCI Bus 0000:06
>> >> c8100000-c8100fff : 0000:06:00.0
>> >
>> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
>> > above look fine, but the top-level host bridge window is missing.
>> >
>> > The c8100000-c91fffff window is a window through the 00:03.0 Root
>> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
>> > memory transactions in the c8100000-c91fffff range from its primary
>> > bus (bus 00) to its secondary bus (bus 02).
>> >
>> > But we don't know how those transactions get to bus 00 in the first
>> > place. The host bridge is the connection between the processor and
>> > bus 00, and it should have a window that encloses c8100000-c91fffff.
>> >
>> > Your dmesg log should have information about this bridge, e.g.,
>> > something like this:
>> >
>> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
>> > PCI host bridge to bus 0000:00
>> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
>> >
>> > and there should be a line in /proc/iomem something like this:
>> >
>> > c8100000-c9ffffff : PCI Bus 0000:00
>> >
>> > Bjorn
>
>> 00000000-00000fff : reserved
>> 00001000-0009abff : System RAM
>> 0009ac00-0009ffff : reserved
>> 000c0000-000c7fff : Video ROM
>> 000c8000-000c8fff : Adapter ROM
>> 000e0000-000fffff : reserved
>> 000f0000-000fffff : System ROM
>> 00100000-78f36fff : System RAM
>> 01000000-01828f14 : Kernel code
>> 01828f15-01f3fb3f : Kernel data
>> 020bb000-021fdfff : Kernel bss
>> 30000000-37ffffff : Crash kernel
>> 78f37000-79839fff : reserved
>> 79116018-79116018 : APEI ERST
>> 7911601c-79116021 : APEI ERST
>> 79116028-79116039 : APEI ERST
>> 79116040-7911604c : APEI ERST
>> 79116050-7911804f : APEI ERST
>> 7983a000-79d4afff : ACPI Non-volatile Storage
>> 79d4b000-8fffffff : reserved
>> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> c6000000-c70fffff : PCI Bus 0000:2f
>> c6000000-c70fffff : PCI Bus 0000:30
>> c6000000-c6ffffff : 0000:30:00.0
>> c7000000-c701ffff : 0000:30:00.0
>> c7100000-c71fffff : PCI Bus 0000:31
>> c7100000-c711ffff : 0000:31:00.1
>> c7100000-c711ffff : igb
>> c7120000-c713ffff : 0000:31:00.0
>> c7120000-c713ffff : igb
>> c7140000-c7143fff : 0000:31:00.1
>> c7140000-c7143fff : igb
>> c7144000-c7147fff : 0000:31:00.0
>> c7144000-c7147fff : igb
>> c7300000-c730ffff : 0000:00:14.0
>> c7300000-c730ffff : xhci-hcd
>> c7310000-c73107ff : 0000:00:1f.2
>> c7310000-c73107ff : ahci
>> c7311000-c73113ff : 0000:00:1d.0
>> c7311000-c73113ff : ehci_hcd
>> c7312000-c73123ff : 0000:00:1a.0
>> c7312000-c73123ff : ehci_hcd
>> c7314000-c731400f : 0000:00:16.1
>> c7315000-c731500f : 0000:00:16.0
>> c7316000-c73167ff : 0000:00:11.4
>> c7316000-c73167ff : ahci
>> c7317000-c7317fff : 0000:00:05.4
>> c7ffc000-c7ffcfff : dmar1
>> c8000000-c8000000 : <BAD>
>> c8000001-c8000001 : <BAD>
>> c8000002-c8000002 : <BAD>
>> c8000003-c8000003 : <BAD>
>> c8100000-c91fffff : PCI Bus 0000:02
>> c8100000-c90fffff : PCI Bus 0000:03
>> c8100000-c90fffff : PCI Bus 0000:04
>> c8100000-c82fffff : PCI Bus 0000:05
>> c8100000-c81fffff : PCI Bus 0000:06
>> c8100000-c8100fff : 0000:06:00.0
>> c8200000-c82fffff : PCI Bus 0000:0a
>> c8200000-c8203fff : 0000:0a:00.0
>> c9100000-c913ffff : 0000:02:00.0
>> fb000000-fb8fffff : PCI Bus 0000:81
>> fb000000-fb8fffff : PCI Bus 0000:82
>> fb000000-fb7fffff : PCI Bus 0000:8c
>> fb000000-fb7fffff : 0000:8c:00.0
>> fb000000-fb7fffff : Plx_Mgr
>> fb800000-fb8fffff : PCI Bus 0000:87
>> fb800000-fb800fff : 0000:87:00.0
>> fb900000-fb900fff : 0000:80:05.4
>> fbffc000-fbffcfff : dmar0
>> fec00000-fecfffff : PNP0003:00
>> fec00000-fec003ff : IOAPIC 0
>> fec01000-fec013ff : IOAPIC 1
>> fec40000-fec403ff : IOAPIC 2
>> fed00000-fed003ff : HPET 0
>> fed00000-fed003ff : PNP0103:00
>> fed12000-fed1200f : pnp 00:01
>> fed12010-fed1201f : pnp 00:01
>> fed1b000-fed1bfff : pnp 00:01
>> fed1c000-fed44fff : reserved
>> fed1c000-fed3ffff : pnp 00:01
>> fed1f410-fed1f414 : iTCO_wdt.0.auto
>> fed45000-fed8bfff : pnp 00:01
>> fee00000-feefffff : pnp 00:01
>> fee00000-fee00fff : Local APIC
>> ff000000-ffffffff : reserved
>> ff000000-ffffffff : pnp 00:01
>> 100000000-47fffffff : System RAM
>> 480000000-4807fffff : PCI Bus 0000:02
>> 480000000-4807fffff : PCI Bus 0000:03
>> 480000000-4807fffff : PCI Bus 0000:04
>> 480000000-4800fffff : PCI Bus 0000:05
>> 480000000-4800fffff : PCI Bus 0000:06
>> 480000000-480003fff : 0000:06:00.0
>> 383fffe00000-383fffefffff : PCI Bus 0000:31
>> 383fffe00000-383fffe1ffff : 0000:31:00.1
>> 383fffe20000-383fffe3ffff : 0000:31:00.1
>> 383fffe40000-383fffe5ffff : 0000:31:00.0
>> 383fffe60000-383fffe7ffff : 0000:31:00.0
>> 383ffff00000-383ffff03fff : 0000:00:04.7
>> 383ffff00000-383ffff03fff : ioatdma
>> 383ffff04000-383ffff07fff : 0000:00:04.6
>> 383ffff04000-383ffff07fff : ioatdma
>> 383ffff08000-383ffff0bfff : 0000:00:04.5
>> 383ffff08000-383ffff0bfff : ioatdma
>> 383ffff0c000-383ffff0ffff : 0000:00:04.4
>> 383ffff0c000-383ffff0ffff : ioatdma
>> 383ffff10000-383ffff13fff : 0000:00:04.3
>> 383ffff10000-383ffff13fff : ioatdma
>> 383ffff14000-383ffff17fff : 0000:00:04.2
>> 383ffff14000-383ffff17fff : ioatdma
>> 383ffff18000-383ffff1bfff : 0000:00:04.1
>> 383ffff18000-383ffff1bfff : ioatdma
>> 383ffff1c000-383ffff1ffff : 0000:00:04.0
>> 383ffff1c000-383ffff1ffff : ioatdma
>> 383ffff21000-383ffff210ff : 0000:00:1f.3
>> 387fffe00000-387fffefffff : PCI Bus 0000:81
>> 387fffe00000-387fffefffff : PCI Bus 0000:82
>> 387fffe00000-387fffefffff : PCI Bus 0000:87
>> 387fffe00000-387fffe03fff : 0000:87:00.0
>> 387ffff00000-387ffff03fff : 0000:80:04.7
>> 387ffff00000-387ffff03fff : ioatdma
>> 387ffff04000-387ffff07fff : 0000:80:04.6
>> 387ffff04000-387ffff07fff : ioatdma
>> 387ffff08000-387ffff0bfff : 0000:80:04.5
>> 387ffff08000-387ffff0bfff : ioatdma
>> 387ffff0c000-387ffff0ffff : 0000:80:04.4
>> 387ffff0c000-387ffff0ffff : ioatdma
>> 387ffff10000-387ffff13fff : 0000:80:04.3
>> 387ffff10000-387ffff13fff : ioatdma
>> 387ffff14000-387ffff17fff : 0000:80:04.2
>> 387ffff14000-387ffff17fff : ioatdma
>> 387ffff18000-387ffff1bfff : 0000:80:04.1
>> 387ffff18000-387ffff1bfff : ioatdma
>> 387ffff1c000-387ffff1ffff : 0000:80:04.0
>> 387ffff1c000-387ffff1ffff : ioatdma
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming
2016-08-15 20:43 ` Kallol Biswas
@ 2016-08-15 21:07 ` Bjorn Helgaas
2016-08-15 21:51 ` Kallol Biswas
0 siblings, 1 reply; 8+ messages in thread
From: Bjorn Helgaas @ 2016-08-15 21:07 UTC (permalink / raw)
To: Kallol Biswas; +Cc: linux-pci
On Mon, Aug 15, 2016 at 01:43:03PM -0700, Kallol Biswas wrote:
> The kernel source code is 4.4. I have not done any modification except
> pre-allocated bus numbers and memory window for a DS port to allow
> successful hot plug operation of a PCIe port.
>
> 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored)
> [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7
> window] (ignored)
> [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff
> window] (ignored)
> [ 0.611924] acpi PNP0A08:00: host bridge window [mem
> 0x000a0000-0x000bffff window] (ignored)
> [ 0.611926] acpi PNP0A08:00: host bridge window [mem
> 0xfedb0000-0xfedb000f window] (ignored)
> [ 0.611928] acpi PNP0A08:00: host bridge window [mem
> 0xfedc0000-0xfedc000f window] (ignored)
> [ 0.611929] acpi PNP0A08:00: host bridge window [mem
> 0x90000000-0xc7ffbfff window] (ignored)
> [ 0.611931] acpi PNP0A08:00: host bridge window [mem
> 0x380000000000-0x383fffffffff window] (ignored)
> [ 0.611932] PCI: root bus 00: using default resources
> [ 0.612217] PCI host bridge to bus 0000:00
> [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
> [ 0.612220] pci_bus 0000:00: root bus resource [mem
> 0x00000000-0x3fffffffffff]
> [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e]
>
> kernel command line:
>
> BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb
> root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro
> pci=realloc,nocrs,assign-busses pcie_ports=native
> pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet
> splash crashkernel=384M-:128M vt.handoff=7
You're booting with "pci=nocrs", which tells Linux to ignore the ACPI
information about host bridge windows. You can't expect resource
allocation and hotplug to work correctly when you're ignoring that.
> On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote:
> >> Thank you for the explanation.
> >>
> >>
> >> Here is the relevant dmesg content:
> >>
> >> [ 0.790457] PCI: root bus ff: using default resources
> >> [ 0.790491] PCI host bridge to bus 0000:ff
> >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
> >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem
> >> 0x00000000-0x3fffffffffff]
> >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
> >>
> >> 0.795379] PCI host bridge to bus 0000:7f
> >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
> >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem
> >> 0x00000000-0x3fffffffffff]
> >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
> >>
> >> 0.809149] PCI host bridge to bus 0000:00
> >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
> >> [ 0.809153] pci_bus 0000:00: root bus resource [mem
> >> 0x00000000-0x3fffffffffff]
> >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
> >
> > This is the host bridge that claims to lead to bus 0000:00.
> >
> > But the I/O and memory resources of all these bridges are wrong. The
> > ranges they show are CPU addresses, not PCI bus addresses, so they
> > cannot overlap.
> >
> > It could be that you're using a non-standard host bridge driver that
> > did not apply the translation between CPU addresses and PCI bus
> > addresses. In ACPI, this is the _TRA property.
> >
> >> [ 0.852071] PCI host bridge to bus 0000:80
> >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
> >> [ 0.852075] pci_bus 0000:80: root bus resource [mem
> >> 0x00000000-0x3fffffffffff]
> >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
> >>
> >> Following lines are from output of /proc/iomem
> >>
> >> c8000000-c8000000 : <BAD>
> >> c8000001-c8000001 : <BAD>
> >> c8000002-c8000002 : <BAD>
> >> c8000003-c8000003 : <BAD>
> >> c8100000-c91fffff : PCI Bus 0000:02
> >> c8100000-c90fffff : PCI Bus 0000:03
> >> c8100000-c90fffff : PCI Bus 0000:04
> >> c8100000-c82fffff : PCI Bus 0000:05
> >> c8100000-c81fffff : PCI Bus 0000:06
> >> c8100000-c8100fff : 0000:06:00.0
> >> c8200000-c82fffff : PCI Bus 0000:0a
> >> c8200000-c8203fff : 0000:0a:00.0
> >> c9100000-c913ffff : 0000:02:00.0
> >> fb000000-fb8fffff : PCI Bus 0000:81
> >> fb000000-fb8fffff : PCI Bus 0000:82
> >> fb000000-fb7fffff : PCI Bus 0000:8c
> >> fb000000-fb7fffff : 0000:8c:00.0
> >> fb000000-fb7fffff : Plx_Mgr
> >> fb800000-fb8fffff : PCI Bus 0000:87
> >> fb800000-fb800fff : 0000:87:00.0
> >> fb900000-fb900fff : 0000:80:05.4
> >>
> >>
> >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
> >> device is available after reboot, and initial PCIe bus enumeration
> >> time.
> >>
> >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
> >> operation is done.
> >>
> >> Attached is the complete /proc/iomem and do not find the line
> >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
> >> PCI Bus 0000:80? BAR access works fine through this bus.
> >>
> >>
> >>
> >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> >> > Hi Kallol,
> >> >
> >> > Sorry I didn't respond to this earlier; I was on vacation for most of
> >> > last week.
> >> >
> >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
> >> >> I am trying to debug an issue that the PCIe memory
> >> >> transactions are not routed through a host bridge.
> >> >>
> >> >> The config transactions work fine. The primary, secondary and
> >> >> subordinate bus numbers look fine for the PCIe tree under the host
> >> >> bridge. The memory windows are programmed fine.
> >> >>
> >> >> Can someone point me to the code that programs the chipset/root port
> >> >> registers those direct processor transactions to a specific host
> >> >> bridge?
> >> >
> >> > It looks like you have an x86 system with ACPI. In that case, we use
> >> > the ACPI pci_root.c host bridge driver. The chipset configuration for
> >> > routing processor memory transactions to host bridges is done by the
> >> > firmware, and the kernel doesn't touch it. There should be
> >> > information in the chipset spec about how to do this, but there's no
> >> > code in Linux itself that does it.
> >> >
> >> >> Cat of /proc/mem is copied below.
> >> >>
> >> >> /root/physmem -p c8100000 -s 4 -r
> >> >> Operation:read to Addr: 0xc8100000 Value: 0
> >> >> ffffffff ffffffff ffffffff ffffffff
> >> >>
> >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
> >> >> external switches/devices does not capture any mem transactions.
> >> >> However, config transactions are visible in the trace.
> >> >
> >> > 00:03.0 is not a host bridge; it's a Root Port, which logically
> >> > connects two PCI buses. A host bridge logically connects a processor
> >> > bus to a PCI bus.
> >> >
> >> >> root@xlink-target:~# cat /proc/iomem
> >> >> 00000000-00000fff : reserved
> >> >> 00001000-0009abff : System RAM
> >> >> 0009ac00-0009ffff : reserved
> >> >> 000c0000-000c7fff : Video ROM
> >> >> 000c8000-000c8fff : Adapter ROM
> >> >> 000e0000-000fffff : reserved
> >> >> 000f0000-000fffff : System ROM
> >> >> 00100000-78f36fff : System RAM
> >> >> 01000000-01828f14 : Kernel code
> >> >> 01828f15-01f3fb3f : Kernel data
> >> >> 020bb000-021fdfff : Kernel bss
> >> >> 30000000-37ffffff : Crash kernel
> >> >> 78f37000-79839fff : reserved
> >> >> 79116018-79116018 : APEI ERST
> >> >> 7911601c-79116021 : APEI ERST
> >> >> 79116028-79116039 : APEI ERST
> >> >> 79116040-7911604c : APEI ERST
> >> >> 79116050-7911804f : APEI ERST
> >> >> 7983a000-79d4afff : ACPI Non-volatile Storage
> >> >> 79d4b000-8fffffff : reserved
> >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> >> >> c6000000-c70fffff : PCI Bus 0000:2f
> >> >> c6000000-c70fffff : PCI Bus 0000:30
> >> >> c6000000-c6ffffff : 0000:30:00.0
> >> >> c7000000-c701ffff : 0000:30:00.0
> >> >> c7100000-c71fffff : PCI Bus 0000:31
> >> >> c7100000-c711ffff : 0000:31:00.1
> >> >> c7100000-c711ffff : igb
> >> >> c7120000-c713ffff : 0000:31:00.0
> >> >> c7120000-c713ffff : igb
> >> >> c7140000-c7143fff : 0000:31:00.1
> >> >> c7140000-c7143fff : igb
> >> >> c7144000-c7147fff : 0000:31:00.0
> >> >> c7144000-c7147fff : igb
> >> >> c7300000-c730ffff : 0000:00:14.0
> >> >> c7300000-c730ffff : xhci-hcd
> >> >> c7310000-c73107ff : 0000:00:1f.2
> >> >> c7310000-c73107ff : ahci
> >> >> c7311000-c73113ff : 0000:00:1d.0
> >> >> c7311000-c73113ff : ehci_hcd
> >> >> c7312000-c73123ff : 0000:00:1a.0
> >> >> c7312000-c73123ff : ehci_hcd
> >> >> c7314000-c731400f : 0000:00:16.1
> >> >> c7315000-c731500f : 0000:00:16.0
> >> >> c7316000-c73167ff : 0000:00:11.4
> >> >> c7316000-c73167ff : ahci
> >> >> c7317000-c7317fff : 0000:00:05.4
> >> >> c7ffc000-c7ffcfff : dmar1
> >> >> c8000000-c8000000 : <BAD>
> >> >> c8000001-c8000001 : <BAD>
> >> >> c8000002-c8000002 : <BAD>
> >> >> c8000003-c8000003 : <BAD>
> >> >
> >> > Interesting, what are these from?
> >> >
> >> >> c8100000-c91fffff : PCI Bus 0000:02
> >> >> c8100000-c90fffff : PCI Bus 0000:03
> >> >> c8100000-c90fffff : PCI Bus 0000:04
> >> >> c8100000-c82fffff : PCI Bus 0000:05
> >> >> c8100000-c81fffff : PCI Bus 0000:06
> >> >> c8100000-c8100fff : 0000:06:00.0
> >> >
> >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
> >> > above look fine, but the top-level host bridge window is missing.
> >> >
> >> > The c8100000-c91fffff window is a window through the 00:03.0 Root
> >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
> >> > memory transactions in the c8100000-c91fffff range from its primary
> >> > bus (bus 00) to its secondary bus (bus 02).
> >> >
> >> > But we don't know how those transactions get to bus 00 in the first
> >> > place. The host bridge is the connection between the processor and
> >> > bus 00, and it should have a window that encloses c8100000-c91fffff.
> >> >
> >> > Your dmesg log should have information about this bridge, e.g.,
> >> > something like this:
> >> >
> >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
> >> > PCI host bridge to bus 0000:00
> >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
> >> >
> >> > and there should be a line in /proc/iomem something like this:
> >> >
> >> > c8100000-c9ffffff : PCI Bus 0000:00
> >> >
> >> > Bjorn
> >
> >> 00000000-00000fff : reserved
> >> 00001000-0009abff : System RAM
> >> 0009ac00-0009ffff : reserved
> >> 000c0000-000c7fff : Video ROM
> >> 000c8000-000c8fff : Adapter ROM
> >> 000e0000-000fffff : reserved
> >> 000f0000-000fffff : System ROM
> >> 00100000-78f36fff : System RAM
> >> 01000000-01828f14 : Kernel code
> >> 01828f15-01f3fb3f : Kernel data
> >> 020bb000-021fdfff : Kernel bss
> >> 30000000-37ffffff : Crash kernel
> >> 78f37000-79839fff : reserved
> >> 79116018-79116018 : APEI ERST
> >> 7911601c-79116021 : APEI ERST
> >> 79116028-79116039 : APEI ERST
> >> 79116040-7911604c : APEI ERST
> >> 79116050-7911804f : APEI ERST
> >> 7983a000-79d4afff : ACPI Non-volatile Storage
> >> 79d4b000-8fffffff : reserved
> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> >> c6000000-c70fffff : PCI Bus 0000:2f
> >> c6000000-c70fffff : PCI Bus 0000:30
> >> c6000000-c6ffffff : 0000:30:00.0
> >> c7000000-c701ffff : 0000:30:00.0
> >> c7100000-c71fffff : PCI Bus 0000:31
> >> c7100000-c711ffff : 0000:31:00.1
> >> c7100000-c711ffff : igb
> >> c7120000-c713ffff : 0000:31:00.0
> >> c7120000-c713ffff : igb
> >> c7140000-c7143fff : 0000:31:00.1
> >> c7140000-c7143fff : igb
> >> c7144000-c7147fff : 0000:31:00.0
> >> c7144000-c7147fff : igb
> >> c7300000-c730ffff : 0000:00:14.0
> >> c7300000-c730ffff : xhci-hcd
> >> c7310000-c73107ff : 0000:00:1f.2
> >> c7310000-c73107ff : ahci
> >> c7311000-c73113ff : 0000:00:1d.0
> >> c7311000-c73113ff : ehci_hcd
> >> c7312000-c73123ff : 0000:00:1a.0
> >> c7312000-c73123ff : ehci_hcd
> >> c7314000-c731400f : 0000:00:16.1
> >> c7315000-c731500f : 0000:00:16.0
> >> c7316000-c73167ff : 0000:00:11.4
> >> c7316000-c73167ff : ahci
> >> c7317000-c7317fff : 0000:00:05.4
> >> c7ffc000-c7ffcfff : dmar1
> >> c8000000-c8000000 : <BAD>
> >> c8000001-c8000001 : <BAD>
> >> c8000002-c8000002 : <BAD>
> >> c8000003-c8000003 : <BAD>
> >> c8100000-c91fffff : PCI Bus 0000:02
> >> c8100000-c90fffff : PCI Bus 0000:03
> >> c8100000-c90fffff : PCI Bus 0000:04
> >> c8100000-c82fffff : PCI Bus 0000:05
> >> c8100000-c81fffff : PCI Bus 0000:06
> >> c8100000-c8100fff : 0000:06:00.0
> >> c8200000-c82fffff : PCI Bus 0000:0a
> >> c8200000-c8203fff : 0000:0a:00.0
> >> c9100000-c913ffff : 0000:02:00.0
> >> fb000000-fb8fffff : PCI Bus 0000:81
> >> fb000000-fb8fffff : PCI Bus 0000:82
> >> fb000000-fb7fffff : PCI Bus 0000:8c
> >> fb000000-fb7fffff : 0000:8c:00.0
> >> fb000000-fb7fffff : Plx_Mgr
> >> fb800000-fb8fffff : PCI Bus 0000:87
> >> fb800000-fb800fff : 0000:87:00.0
> >> fb900000-fb900fff : 0000:80:05.4
> >> fbffc000-fbffcfff : dmar0
> >> fec00000-fecfffff : PNP0003:00
> >> fec00000-fec003ff : IOAPIC 0
> >> fec01000-fec013ff : IOAPIC 1
> >> fec40000-fec403ff : IOAPIC 2
> >> fed00000-fed003ff : HPET 0
> >> fed00000-fed003ff : PNP0103:00
> >> fed12000-fed1200f : pnp 00:01
> >> fed12010-fed1201f : pnp 00:01
> >> fed1b000-fed1bfff : pnp 00:01
> >> fed1c000-fed44fff : reserved
> >> fed1c000-fed3ffff : pnp 00:01
> >> fed1f410-fed1f414 : iTCO_wdt.0.auto
> >> fed45000-fed8bfff : pnp 00:01
> >> fee00000-feefffff : pnp 00:01
> >> fee00000-fee00fff : Local APIC
> >> ff000000-ffffffff : reserved
> >> ff000000-ffffffff : pnp 00:01
> >> 100000000-47fffffff : System RAM
> >> 480000000-4807fffff : PCI Bus 0000:02
> >> 480000000-4807fffff : PCI Bus 0000:03
> >> 480000000-4807fffff : PCI Bus 0000:04
> >> 480000000-4800fffff : PCI Bus 0000:05
> >> 480000000-4800fffff : PCI Bus 0000:06
> >> 480000000-480003fff : 0000:06:00.0
> >> 383fffe00000-383fffefffff : PCI Bus 0000:31
> >> 383fffe00000-383fffe1ffff : 0000:31:00.1
> >> 383fffe20000-383fffe3ffff : 0000:31:00.1
> >> 383fffe40000-383fffe5ffff : 0000:31:00.0
> >> 383fffe60000-383fffe7ffff : 0000:31:00.0
> >> 383ffff00000-383ffff03fff : 0000:00:04.7
> >> 383ffff00000-383ffff03fff : ioatdma
> >> 383ffff04000-383ffff07fff : 0000:00:04.6
> >> 383ffff04000-383ffff07fff : ioatdma
> >> 383ffff08000-383ffff0bfff : 0000:00:04.5
> >> 383ffff08000-383ffff0bfff : ioatdma
> >> 383ffff0c000-383ffff0ffff : 0000:00:04.4
> >> 383ffff0c000-383ffff0ffff : ioatdma
> >> 383ffff10000-383ffff13fff : 0000:00:04.3
> >> 383ffff10000-383ffff13fff : ioatdma
> >> 383ffff14000-383ffff17fff : 0000:00:04.2
> >> 383ffff14000-383ffff17fff : ioatdma
> >> 383ffff18000-383ffff1bfff : 0000:00:04.1
> >> 383ffff18000-383ffff1bfff : ioatdma
> >> 383ffff1c000-383ffff1ffff : 0000:00:04.0
> >> 383ffff1c000-383ffff1ffff : ioatdma
> >> 383ffff21000-383ffff210ff : 0000:00:1f.3
> >> 387fffe00000-387fffefffff : PCI Bus 0000:81
> >> 387fffe00000-387fffefffff : PCI Bus 0000:82
> >> 387fffe00000-387fffefffff : PCI Bus 0000:87
> >> 387fffe00000-387fffe03fff : 0000:87:00.0
> >> 387ffff00000-387ffff03fff : 0000:80:04.7
> >> 387ffff00000-387ffff03fff : ioatdma
> >> 387ffff04000-387ffff07fff : 0000:80:04.6
> >> 387ffff04000-387ffff07fff : ioatdma
> >> 387ffff08000-387ffff0bfff : 0000:80:04.5
> >> 387ffff08000-387ffff0bfff : ioatdma
> >> 387ffff0c000-387ffff0ffff : 0000:80:04.4
> >> 387ffff0c000-387ffff0ffff : ioatdma
> >> 387ffff10000-387ffff13fff : 0000:80:04.3
> >> 387ffff10000-387ffff13fff : ioatdma
> >> 387ffff14000-387ffff17fff : 0000:80:04.2
> >> 387ffff14000-387ffff17fff : ioatdma
> >> 387ffff18000-387ffff1bfff : 0000:80:04.1
> >> 387ffff18000-387ffff1bfff : ioatdma
> >> 387ffff1c000-387ffff1ffff : 0000:80:04.0
> >> 387ffff1c000-387ffff1ffff : ioatdma
> >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming
2016-08-15 21:07 ` Bjorn Helgaas
@ 2016-08-15 21:51 ` Kallol Biswas
2016-08-15 21:56 ` Kallol Biswas
0 siblings, 1 reply; 8+ messages in thread
From: Kallol Biswas @ 2016-08-15 21:51 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: linux-pci
[-- Attachment #1: Type: text/plain, Size: 17899 bytes --]
Hello Bjorn,
Now memory access works: The following window has been
setup properly:
*90000000-c7ffbfff : PCI Bus 0000:00*
9d4b000-8fffffff : reserved
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
90000000-c7ffbfff : PCI Bus 0000:00
90000000-910fffff : PCI Bus 0000:02
90000000-90ffffff : PCI Bus 0000:03
90000000-90ffffff : PCI Bus 0000:04
90000000-901fffff : PCI Bus 0000:05
90000000-900fffff : PCI Bus 0000:06
90000000-90000fff : 0000:06:00.0
90000000-90000fff : r8169
90100000-901fffff : PCI Bus 0000:0a
90100000-90103fff : 0000:0a:00.0
91000000-9103ffff : 0000:02:00.0
c6000000-c70fffff : PCI Bus 0000:2f
Thank you very much. nocrs was creating the problem.
Just FYI, my setup is a regular supermicro PC, running Ubuntu 16.04 and my
modified 4.4 kernel. I am trying to bring up devices via a PCIe fabric path.
On Mon, Aug 15, 2016 at 2:07 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
> On Mon, Aug 15, 2016 at 01:43:03PM -0700, Kallol Biswas wrote:
> > The kernel source code is 4.4. I have not done any modification except
> > pre-allocated bus numbers and memory window for a DS port to allow
> > successful hot plug operation of a PCIe port.
> >
> > 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff]
> (ignored)
> > [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7
> > window] (ignored)
> > [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff
> > window] (ignored)
> > [ 0.611924] acpi PNP0A08:00: host bridge window [mem
> > 0x000a0000-0x000bffff window] (ignored)
> > [ 0.611926] acpi PNP0A08:00: host bridge window [mem
> > 0xfedb0000-0xfedb000f window] (ignored)
> > [ 0.611928] acpi PNP0A08:00: host bridge window [mem
> > 0xfedc0000-0xfedc000f window] (ignored)
> > [ 0.611929] acpi PNP0A08:00: host bridge window [mem
> > 0x90000000-0xc7ffbfff window] (ignored)
> > [ 0.611931] acpi PNP0A08:00: host bridge window [mem
> > 0x380000000000-0x383fffffffff window] (ignored)
> > [ 0.611932] PCI: root bus 00: using default resources
> > [ 0.612217] PCI host bridge to bus 0000:00
> > [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
> > [ 0.612220] pci_bus 0000:00: root bus resource [mem
> > 0x00000000-0x3fffffffffff]
> > [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e]
> >
> > kernel command line:
> >
> > BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb
> > root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro
> > pci=realloc,nocrs,assign-busses pcie_ports=native
> > pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet
> > splash crashkernel=384M-:128M vt.handoff=7
>
> You're booting with "pci=nocrs", which tells Linux to ignore the ACPI
> information about host bridge windows. You can't expect resource
> allocation and hotplug to work correctly when you're ignoring that.
>
> > On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org>
> wrote:
> > > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote:
> > >> Thank you for the explanation.
> > >>
> > >>
> > >> Here is the relevant dmesg content:
> > >>
> > >> [ 0.790457] PCI: root bus ff: using default resources
> > >> [ 0.790491] PCI host bridge to bus 0000:ff
> > >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
> > >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem
> > >> 0x00000000-0x3fffffffffff]
> > >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
> > >>
> > >> 0.795379] PCI host bridge to bus 0000:7f
> > >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
> > >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem
> > >> 0x00000000-0x3fffffffffff]
> > >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
> > >>
> > >> 0.809149] PCI host bridge to bus 0000:00
> > >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
> > >> [ 0.809153] pci_bus 0000:00: root bus resource [mem
> > >> 0x00000000-0x3fffffffffff]
> > >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
> > >
> > > This is the host bridge that claims to lead to bus 0000:00.
> > >
> > > But the I/O and memory resources of all these bridges are wrong. The
> > > ranges they show are CPU addresses, not PCI bus addresses, so they
> > > cannot overlap.
> > >
> > > It could be that you're using a non-standard host bridge driver that
> > > did not apply the translation between CPU addresses and PCI bus
> > > addresses. In ACPI, this is the _TRA property.
> > >
> > >> [ 0.852071] PCI host bridge to bus 0000:80
> > >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
> > >> [ 0.852075] pci_bus 0000:80: root bus resource [mem
> > >> 0x00000000-0x3fffffffffff]
> > >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
> > >>
> > >> Following lines are from output of /proc/iomem
> > >>
> > >> c8000000-c8000000 : <BAD>
> > >> c8000001-c8000001 : <BAD>
> > >> c8000002-c8000002 : <BAD>
> > >> c8000003-c8000003 : <BAD>
> > >> c8100000-c91fffff : PCI Bus 0000:02
> > >> c8100000-c90fffff : PCI Bus 0000:03
> > >> c8100000-c90fffff : PCI Bus 0000:04
> > >> c8100000-c82fffff : PCI Bus 0000:05
> > >> c8100000-c81fffff : PCI Bus 0000:06
> > >> c8100000-c8100fff : 0000:06:00.0
> > >> c8200000-c82fffff : PCI Bus 0000:0a
> > >> c8200000-c8203fff : 0000:0a:00.0
> > >> c9100000-c913ffff : 0000:02:00.0
> > >> fb000000-fb8fffff : PCI Bus 0000:81
> > >> fb000000-fb8fffff : PCI Bus 0000:82
> > >> fb000000-fb7fffff : PCI Bus 0000:8c
> > >> fb000000-fb7fffff : 0000:8c:00.0
> > >> fb000000-fb7fffff : Plx_Mgr
> > >> fb800000-fb8fffff : PCI Bus 0000:87
> > >> fb800000-fb800fff : 0000:87:00.0
> > >> fb900000-fb900fff : 0000:80:05.4
> > >>
> > >>
> > >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
> > >> device is available after reboot, and initial PCIe bus enumeration
> > >> time.
> > >>
> > >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
> > >> operation is done.
> > >>
> > >> Attached is the complete /proc/iomem and do not find the line
> > >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
> > >> PCI Bus 0000:80? BAR access works fine through this bus.
> > >>
> > >>
> > >>
> > >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org>
> wrote:
> > >> > Hi Kallol,
> > >> >
> > >> > Sorry I didn't respond to this earlier; I was on vacation for most
> of
> > >> > last week.
> > >> >
> > >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
> > >> >> I am trying to debug an issue that the PCIe memory
> > >> >> transactions are not routed through a host bridge.
> > >> >>
> > >> >> The config transactions work fine. The primary, secondary and
> > >> >> subordinate bus numbers look fine for the PCIe tree under the host
> > >> >> bridge. The memory windows are programmed fine.
> > >> >>
> > >> >> Can someone point me to the code that programs the chipset/root
> port
> > >> >> registers those direct processor transactions to a specific host
> > >> >> bridge?
> > >> >
> > >> > It looks like you have an x86 system with ACPI. In that case, we
> use
> > >> > the ACPI pci_root.c host bridge driver. The chipset configuration
> for
> > >> > routing processor memory transactions to host bridges is done by the
> > >> > firmware, and the kernel doesn't touch it. There should be
> > >> > information in the chipset spec about how to do this, but there's no
> > >> > code in Linux itself that does it.
> > >> >
> > >> >> Cat of /proc/mem is copied below.
> > >> >>
> > >> >> /root/physmem -p c8100000 -s 4 -r
> > >> >> Operation:read to Addr: 0xc8100000 Value: 0
> > >> >> ffffffff ffffffff ffffffff ffffffff
> > >> >>
> > >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0)
> and
> > >> >> external switches/devices does not capture any mem transactions.
> > >> >> However, config transactions are visible in the trace.
> > >> >
> > >> > 00:03.0 is not a host bridge; it's a Root Port, which logically
> > >> > connects two PCI buses. A host bridge logically connects a
> processor
> > >> > bus to a PCI bus.
> > >> >
> > >> >> root@xlink-target:~# cat /proc/iomem
> > >> >> 00000000-00000fff : reserved
> > >> >> 00001000-0009abff : System RAM
> > >> >> 0009ac00-0009ffff : reserved
> > >> >> 000c0000-000c7fff : Video ROM
> > >> >> 000c8000-000c8fff : Adapter ROM
> > >> >> 000e0000-000fffff : reserved
> > >> >> 000f0000-000fffff : System ROM
> > >> >> 00100000-78f36fff : System RAM
> > >> >> 01000000-01828f14 : Kernel code
> > >> >> 01828f15-01f3fb3f : Kernel data
> > >> >> 020bb000-021fdfff : Kernel bss
> > >> >> 30000000-37ffffff : Crash kernel
> > >> >> 78f37000-79839fff : reserved
> > >> >> 79116018-79116018 : APEI ERST
> > >> >> 7911601c-79116021 : APEI ERST
> > >> >> 79116028-79116039 : APEI ERST
> > >> >> 79116040-7911604c : APEI ERST
> > >> >> 79116050-7911804f : APEI ERST
> > >> >> 7983a000-79d4afff : ACPI Non-volatile Storage
> > >> >> 79d4b000-8fffffff : reserved
> > >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> > >> >> c6000000-c70fffff : PCI Bus 0000:2f
> > >> >> c6000000-c70fffff : PCI Bus 0000:30
> > >> >> c6000000-c6ffffff : 0000:30:00.0
> > >> >> c7000000-c701ffff : 0000:30:00.0
> > >> >> c7100000-c71fffff : PCI Bus 0000:31
> > >> >> c7100000-c711ffff : 0000:31:00.1
> > >> >> c7100000-c711ffff : igb
> > >> >> c7120000-c713ffff : 0000:31:00.0
> > >> >> c7120000-c713ffff : igb
> > >> >> c7140000-c7143fff : 0000:31:00.1
> > >> >> c7140000-c7143fff : igb
> > >> >> c7144000-c7147fff : 0000:31:00.0
> > >> >> c7144000-c7147fff : igb
> > >> >> c7300000-c730ffff : 0000:00:14.0
> > >> >> c7300000-c730ffff : xhci-hcd
> > >> >> c7310000-c73107ff : 0000:00:1f.2
> > >> >> c7310000-c73107ff : ahci
> > >> >> c7311000-c73113ff : 0000:00:1d.0
> > >> >> c7311000-c73113ff : ehci_hcd
> > >> >> c7312000-c73123ff : 0000:00:1a.0
> > >> >> c7312000-c73123ff : ehci_hcd
> > >> >> c7314000-c731400f : 0000:00:16.1
> > >> >> c7315000-c731500f : 0000:00:16.0
> > >> >> c7316000-c73167ff : 0000:00:11.4
> > >> >> c7316000-c73167ff : ahci
> > >> >> c7317000-c7317fff : 0000:00:05.4
> > >> >> c7ffc000-c7ffcfff : dmar1
> > >> >> c8000000-c8000000 : <BAD>
> > >> >> c8000001-c8000001 : <BAD>
> > >> >> c8000002-c8000002 : <BAD>
> > >> >> c8000003-c8000003 : <BAD>
> > >> >
> > >> > Interesting, what are these from?
> > >> >
> > >> >> c8100000-c91fffff : PCI Bus 0000:02
> > >> >> c8100000-c90fffff : PCI Bus 0000:03
> > >> >> c8100000-c90fffff : PCI Bus 0000:04
> > >> >> c8100000-c82fffff : PCI Bus 0000:05
> > >> >> c8100000-c81fffff : PCI Bus 0000:06
> > >> >> c8100000-c8100fff : 0000:06:00.0
> > >> >
> > >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
> > >> > above look fine, but the top-level host bridge window is missing.
> > >> >
> > >> > The c8100000-c91fffff window is a window through the 00:03.0 Root
> > >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
> > >> > memory transactions in the c8100000-c91fffff range from its primary
> > >> > bus (bus 00) to its secondary bus (bus 02).
> > >> >
> > >> > But we don't know how those transactions get to bus 00 in the first
> > >> > place. The host bridge is the connection between the processor and
> > >> > bus 00, and it should have a window that encloses c8100000-c91fffff.
> > >> >
> > >> > Your dmesg log should have information about this bridge, e.g.,
> > >> > something like this:
> > >> >
> > >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
> > >> > PCI host bridge to bus 0000:00
> > >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff
> window]
> > >> >
> > >> > and there should be a line in /proc/iomem something like this:
> > >> >
> > >> > c8100000-c9ffffff : PCI Bus 0000:00
> > >> >
> > >> > Bjorn
> > >
> > >> 00000000-00000fff : reserved
> > >> 00001000-0009abff : System RAM
> > >> 0009ac00-0009ffff : reserved
> > >> 000c0000-000c7fff : Video ROM
> > >> 000c8000-000c8fff : Adapter ROM
> > >> 000e0000-000fffff : reserved
> > >> 000f0000-000fffff : System ROM
> > >> 00100000-78f36fff : System RAM
> > >> 01000000-01828f14 : Kernel code
> > >> 01828f15-01f3fb3f : Kernel data
> > >> 020bb000-021fdfff : Kernel bss
> > >> 30000000-37ffffff : Crash kernel
> > >> 78f37000-79839fff : reserved
> > >> 79116018-79116018 : APEI ERST
> > >> 7911601c-79116021 : APEI ERST
> > >> 79116028-79116039 : APEI ERST
> > >> 79116040-7911604c : APEI ERST
> > >> 79116050-7911804f : APEI ERST
> > >> 7983a000-79d4afff : ACPI Non-volatile Storage
> > >> 79d4b000-8fffffff : reserved
> > >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> > >> c6000000-c70fffff : PCI Bus 0000:2f
> > >> c6000000-c70fffff : PCI Bus 0000:30
> > >> c6000000-c6ffffff : 0000:30:00.0
> > >> c7000000-c701ffff : 0000:30:00.0
> > >> c7100000-c71fffff : PCI Bus 0000:31
> > >> c7100000-c711ffff : 0000:31:00.1
> > >> c7100000-c711ffff : igb
> > >> c7120000-c713ffff : 0000:31:00.0
> > >> c7120000-c713ffff : igb
> > >> c7140000-c7143fff : 0000:31:00.1
> > >> c7140000-c7143fff : igb
> > >> c7144000-c7147fff : 0000:31:00.0
> > >> c7144000-c7147fff : igb
> > >> c7300000-c730ffff : 0000:00:14.0
> > >> c7300000-c730ffff : xhci-hcd
> > >> c7310000-c73107ff : 0000:00:1f.2
> > >> c7310000-c73107ff : ahci
> > >> c7311000-c73113ff : 0000:00:1d.0
> > >> c7311000-c73113ff : ehci_hcd
> > >> c7312000-c73123ff : 0000:00:1a.0
> > >> c7312000-c73123ff : ehci_hcd
> > >> c7314000-c731400f : 0000:00:16.1
> > >> c7315000-c731500f : 0000:00:16.0
> > >> c7316000-c73167ff : 0000:00:11.4
> > >> c7316000-c73167ff : ahci
> > >> c7317000-c7317fff : 0000:00:05.4
> > >> c7ffc000-c7ffcfff : dmar1
> > >> c8000000-c8000000 : <BAD>
> > >> c8000001-c8000001 : <BAD>
> > >> c8000002-c8000002 : <BAD>
> > >> c8000003-c8000003 : <BAD>
> > >> c8100000-c91fffff : PCI Bus 0000:02
> > >> c8100000-c90fffff : PCI Bus 0000:03
> > >> c8100000-c90fffff : PCI Bus 0000:04
> > >> c8100000-c82fffff : PCI Bus 0000:05
> > >> c8100000-c81fffff : PCI Bus 0000:06
> > >> c8100000-c8100fff : 0000:06:00.0
> > >> c8200000-c82fffff : PCI Bus 0000:0a
> > >> c8200000-c8203fff : 0000:0a:00.0
> > >> c9100000-c913ffff : 0000:02:00.0
> > >> fb000000-fb8fffff : PCI Bus 0000:81
> > >> fb000000-fb8fffff : PCI Bus 0000:82
> > >> fb000000-fb7fffff : PCI Bus 0000:8c
> > >> fb000000-fb7fffff : 0000:8c:00.0
> > >> fb000000-fb7fffff : Plx_Mgr
> > >> fb800000-fb8fffff : PCI Bus 0000:87
> > >> fb800000-fb800fff : 0000:87:00.0
> > >> fb900000-fb900fff : 0000:80:05.4
> > >> fbffc000-fbffcfff : dmar0
> > >> fec00000-fecfffff : PNP0003:00
> > >> fec00000-fec003ff : IOAPIC 0
> > >> fec01000-fec013ff : IOAPIC 1
> > >> fec40000-fec403ff : IOAPIC 2
> > >> fed00000-fed003ff : HPET 0
> > >> fed00000-fed003ff : PNP0103:00
> > >> fed12000-fed1200f : pnp 00:01
> > >> fed12010-fed1201f : pnp 00:01
> > >> fed1b000-fed1bfff : pnp 00:01
> > >> fed1c000-fed44fff : reserved
> > >> fed1c000-fed3ffff : pnp 00:01
> > >> fed1f410-fed1f414 : iTCO_wdt.0.auto
> > >> fed45000-fed8bfff : pnp 00:01
> > >> fee00000-feefffff : pnp 00:01
> > >> fee00000-fee00fff : Local APIC
> > >> ff000000-ffffffff : reserved
> > >> ff000000-ffffffff : pnp 00:01
> > >> 100000000-47fffffff : System RAM
> > >> 480000000-4807fffff : PCI Bus 0000:02
> > >> 480000000-4807fffff : PCI Bus 0000:03
> > >> 480000000-4807fffff : PCI Bus 0000:04
> > >> 480000000-4800fffff : PCI Bus 0000:05
> > >> 480000000-4800fffff : PCI Bus 0000:06
> > >> 480000000-480003fff : 0000:06:00.0
> > >> 383fffe00000-383fffefffff : PCI Bus 0000:31
> > >> 383fffe00000-383fffe1ffff : 0000:31:00.1
> > >> 383fffe20000-383fffe3ffff : 0000:31:00.1
> > >> 383fffe40000-383fffe5ffff : 0000:31:00.0
> > >> 383fffe60000-383fffe7ffff : 0000:31:00.0
> > >> 383ffff00000-383ffff03fff : 0000:00:04.7
> > >> 383ffff00000-383ffff03fff : ioatdma
> > >> 383ffff04000-383ffff07fff : 0000:00:04.6
> > >> 383ffff04000-383ffff07fff : ioatdma
> > >> 383ffff08000-383ffff0bfff : 0000:00:04.5
> > >> 383ffff08000-383ffff0bfff : ioatdma
> > >> 383ffff0c000-383ffff0ffff : 0000:00:04.4
> > >> 383ffff0c000-383ffff0ffff : ioatdma
> > >> 383ffff10000-383ffff13fff : 0000:00:04.3
> > >> 383ffff10000-383ffff13fff : ioatdma
> > >> 383ffff14000-383ffff17fff : 0000:00:04.2
> > >> 383ffff14000-383ffff17fff : ioatdma
> > >> 383ffff18000-383ffff1bfff : 0000:00:04.1
> > >> 383ffff18000-383ffff1bfff : ioatdma
> > >> 383ffff1c000-383ffff1ffff : 0000:00:04.0
> > >> 383ffff1c000-383ffff1ffff : ioatdma
> > >> 383ffff21000-383ffff210ff : 0000:00:1f.3
> > >> 387fffe00000-387fffefffff : PCI Bus 0000:81
> > >> 387fffe00000-387fffefffff : PCI Bus 0000:82
> > >> 387fffe00000-387fffefffff : PCI Bus 0000:87
> > >> 387fffe00000-387fffe03fff : 0000:87:00.0
> > >> 387ffff00000-387ffff03fff : 0000:80:04.7
> > >> 387ffff00000-387ffff03fff : ioatdma
> > >> 387ffff04000-387ffff07fff : 0000:80:04.6
> > >> 387ffff04000-387ffff07fff : ioatdma
> > >> 387ffff08000-387ffff0bfff : 0000:80:04.5
> > >> 387ffff08000-387ffff0bfff : ioatdma
> > >> 387ffff0c000-387ffff0ffff : 0000:80:04.4
> > >> 387ffff0c000-387ffff0ffff : ioatdma
> > >> 387ffff10000-387ffff13fff : 0000:80:04.3
> > >> 387ffff10000-387ffff13fff : ioatdma
> > >> 387ffff14000-387ffff17fff : 0000:80:04.2
> > >> 387ffff14000-387ffff17fff : ioatdma
> > >> 387ffff18000-387ffff1bfff : 0000:80:04.1
> > >> 387ffff18000-387ffff1bfff : ioatdma
> > >> 387ffff1c000-387ffff1ffff : 0000:80:04.0
> > >> 387ffff1c000-387ffff1ffff : ioatdma
> > >
>
[-- Attachment #2: Type: text/html, Size: 23991 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Fwd: PCIe root port programming
2016-08-15 21:51 ` Kallol Biswas
@ 2016-08-15 21:56 ` Kallol Biswas
0 siblings, 0 replies; 8+ messages in thread
From: Kallol Biswas @ 2016-08-15 21:56 UTC (permalink / raw)
To: Bjorn Helgaas; +Cc: linux-pci
> Hello Bjorn,
> Now memory access works: The following window has been setup properly:
> 90000000-c7ffbfff : PCI Bus 0000:00
>
> 9d4b000-8fffffff : reserved
> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
> 90000000-c7ffbfff : PCI Bus 0000:00
> 90000000-910fffff : PCI Bus 0000:02
> 90000000-90ffffff : PCI Bus 0000:03
> 90000000-90ffffff : PCI Bus 0000:04
> 90000000-901fffff : PCI Bus 0000:05
> 90000000-900fffff : PCI Bus 0000:06
> 90000000-90000fff : 0000:06:00.0
> 90000000-90000fff : r8169
> 90100000-901fffff : PCI Bus 0000:0a
> 90100000-90103fff : 0000:0a:00.0
> 91000000-9103ffff : 0000:02:00.0
> c6000000-c70fffff : PCI Bus 0000:2f
>
> Thank you very much. nocrs was creating the problem.
>
> Just FYI, my setup is a regular supermicro PC, running Ubuntu 16.04 and my modified 4.4 kernel. I am trying to bring up devices via a PCIe fabric path.
>
>
> On Mon, Aug 15, 2016 at 2:07 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>>
>> On Mon, Aug 15, 2016 at 01:43:03PM -0700, Kallol Biswas wrote:
>> > The kernel source code is 4.4. I have not done any modification except
>> > pre-allocated bus numbers and memory window for a DS port to allow
>> > successful hot plug operation of a PCIe port.
>> >
>> > 0.611919] acpi PNP0A08:00: host bridge window [io 0x0cf8-0x0cff] (ignored)
>> > [ 0.611921] acpi PNP0A08:00: host bridge window [io 0x0000-0x0cf7
>> > window] (ignored)
>> > [ 0.611922] acpi PNP0A08:00: host bridge window [io 0x1000-0x7fff
>> > window] (ignored)
>> > [ 0.611924] acpi PNP0A08:00: host bridge window [mem
>> > 0x000a0000-0x000bffff window] (ignored)
>> > [ 0.611926] acpi PNP0A08:00: host bridge window [mem
>> > 0xfedb0000-0xfedb000f window] (ignored)
>> > [ 0.611928] acpi PNP0A08:00: host bridge window [mem
>> > 0xfedc0000-0xfedc000f window] (ignored)
>> > [ 0.611929] acpi PNP0A08:00: host bridge window [mem
>> > 0x90000000-0xc7ffbfff window] (ignored)
>> > [ 0.611931] acpi PNP0A08:00: host bridge window [mem
>> > 0x380000000000-0x383fffffffff window] (ignored)
>> > [ 0.611932] PCI: root bus 00: using default resources
>> > [ 0.612217] PCI host bridge to bus 0000:00
>> > [ 0.612219] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
>> > [ 0.612220] pci_bus 0000:00: root bus resource [mem
>> > 0x00000000-0x3fffffffffff]
>> > [ 0.612222] pci_bus 0000:00: root bus resource [bus 00-7e]
>> >
>> > kernel command line:
>> >
>> > BOOT_IMAGE=/boot/vmlinuz-4.4.8-kb
>> > root=UUID=c6f326f6-7cc4-49f5-a3be-c511566165b4 ro
>> > pci=realloc,nocrs,assign-busses pcie_ports=native
>> > pciehp.pciehp_debug=1 pci=hpmemsize=0M console=ttyS1,115200n8 quiet
>> > splash crashkernel=384M-:128M vt.handoff=7
>>
>> You're booting with "pci=nocrs", which tells Linux to ignore the ACPI
>> information about host bridge windows. You can't expect resource
>> allocation and hotplug to work correctly when you're ignoring that.
>>
>> > On Mon, Aug 15, 2016 at 12:06 PM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>> > > On Mon, Aug 15, 2016 at 11:39:56AM -0700, Kallol Biswas wrote:
>> > >> Thank you for the explanation.
>> > >>
>> > >>
>> > >> Here is the relevant dmesg content:
>> > >>
>> > >> [ 0.790457] PCI: root bus ff: using default resources
>> > >> [ 0.790491] PCI host bridge to bus 0000:ff
>> > >> [ 0.790493] pci_bus 0000:ff: root bus resource [io 0x0000-0xffff]
>> > >> [ 0.790495] pci_bus 0000:ff: root bus resource [mem
>> > >> 0x00000000-0x3fffffffffff]
>> > >> [ 0.790497] pci_bus 0000:ff: root bus resource [bus ff]
>> > >>
>> > >> 0.795379] PCI host bridge to bus 0000:7f
>> > >> [ 0.795381] pci_bus 0000:7f: root bus resource [io 0x0000-0xffff]
>> > >> [ 0.795383] pci_bus 0000:7f: root bus resource [mem
>> > >> 0x00000000-0x3fffffffffff]
>> > >> [ 0.795384] pci_bus 0000:7f: root bus resource [bus 7f]
>> > >>
>> > >> 0.809149] PCI host bridge to bus 0000:00
>> > >> [ 0.809151] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
>> > >> [ 0.809153] pci_bus 0000:00: root bus resource [mem
>> > >> 0x00000000-0x3fffffffffff]
>> > >> [ 0.809155] pci_bus 0000:00: root bus resource [bus 00-7e]
>> > >
>> > > This is the host bridge that claims to lead to bus 0000:00.
>> > >
>> > > But the I/O and memory resources of all these bridges are wrong. The
>> > > ranges they show are CPU addresses, not PCI bus addresses, so they
>> > > cannot overlap.
>> > >
>> > > It could be that you're using a non-standard host bridge driver that
>> > > did not apply the translation between CPU addresses and PCI bus
>> > > addresses. In ACPI, this is the _TRA property.
>> > >
>> > >> [ 0.852071] PCI host bridge to bus 0000:80
>> > >> [ 0.852073] pci_bus 0000:80: root bus resource [io 0x0000-0xffff]
>> > >> [ 0.852075] pci_bus 0000:80: root bus resource [mem
>> > >> 0x00000000-0x3fffffffffff]
>> > >> [ 0.852077] pci_bus 0000:80: root bus resource [bus 80-fe]
>> > >>
>> > >> Following lines are from output of /proc/iomem
>> > >>
>> > >> c8000000-c8000000 : <BAD>
>> > >> c8000001-c8000001 : <BAD>
>> > >> c8000002-c8000002 : <BAD>
>> > >> c8000003-c8000003 : <BAD>
>> > >> c8100000-c91fffff : PCI Bus 0000:02
>> > >> c8100000-c90fffff : PCI Bus 0000:03
>> > >> c8100000-c90fffff : PCI Bus 0000:04
>> > >> c8100000-c82fffff : PCI Bus 0000:05
>> > >> c8100000-c81fffff : PCI Bus 0000:06
>> > >> c8100000-c8100fff : 0000:06:00.0
>> > >> c8200000-c82fffff : PCI Bus 0000:0a
>> > >> c8200000-c8203fff : 0000:0a:00.0
>> > >> c9100000-c913ffff : 0000:02:00.0
>> > >> fb000000-fb8fffff : PCI Bus 0000:81
>> > >> fb000000-fb8fffff : PCI Bus 0000:82
>> > >> fb000000-fb7fffff : PCI Bus 0000:8c
>> > >> fb000000-fb7fffff : 0000:8c:00.0
>> > >> fb000000-fb7fffff : Plx_Mgr
>> > >> fb800000-fb8fffff : PCI Bus 0000:87
>> > >> fb800000-fb800fff : 0000:87:00.0
>> > >> fb900000-fb900fff : 0000:80:05.4
>> > >>
>> > >>
>> > >> I can read the device BAR via fb800000-fb800fff : 0000:87:00.0. This
>> > >> device is available after reboot, and initial PCIe bus enumeration
>> > >> time.
>> > >>
>> > >> The device c8100000-c8100fff : 0000:06:00.0 shows up after a hotplug
>> > >> operation is done.
>> > >>
>> > >> Attached is the complete /proc/iomem and do not find the line
>> > >> c8100000-c9ffffff : PCI Bus 0000:00, but does it exist for
>> > >> PCI Bus 0000:80? BAR access works fine through this bus.
>> > >>
>> > >>
>> > >>
>> > >> On Mon, Aug 15, 2016 at 10:24 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:
>> > >> > Hi Kallol,
>> > >> >
>> > >> > Sorry I didn't respond to this earlier; I was on vacation for most of
>> > >> > last week.
>> > >> >
>> > >> > On Fri, Aug 12, 2016 at 02:14:59PM -0700, Kallol Biswas wrote:
>> > >> >> I am trying to debug an issue that the PCIe memory
>> > >> >> transactions are not routed through a host bridge.
>> > >> >>
>> > >> >> The config transactions work fine. The primary, secondary and
>> > >> >> subordinate bus numbers look fine for the PCIe tree under the host
>> > >> >> bridge. The memory windows are programmed fine.
>> > >> >>
>> > >> >> Can someone point me to the code that programs the chipset/root port
>> > >> >> registers those direct processor transactions to a specific host
>> > >> >> bridge?
>> > >> >
>> > >> > It looks like you have an x86 system with ACPI. In that case, we use
>> > >> > the ACPI pci_root.c host bridge driver. The chipset configuration for
>> > >> > routing processor memory transactions to host bridges is done by the
>> > >> > firmware, and the kernel doesn't touch it. There should be
>> > >> > information in the chipset spec about how to do this, but there's no
>> > >> > code in Linux itself that does it.
>> > >> >
>> > >> >> Cat of /proc/mem is copied below.
>> > >> >>
>> > >> >> /root/physmem -p c8100000 -s 4 -r
>> > >> >> Operation:read to Addr: 0xc8100000 Value: 0
>> > >> >> ffffffff ffffffff ffffffff ffffffff
>> > >> >>
>> > >> >> The PCIe analyzer sitting between host bridge (0:3.0 and 02:00.0) and
>> > >> >> external switches/devices does not capture any mem transactions.
>> > >> >> However, config transactions are visible in the trace.
>> > >> >
>> > >> > 00:03.0 is not a host bridge; it's a Root Port, which logically
>> > >> > connects two PCI buses. A host bridge logically connects a processor
>> > >> > bus to a PCI bus.
>> > >> >
>> > >> >> root@xlink-target:~# cat /proc/iomem
>> > >> >> 00000000-00000fff : reserved
>> > >> >> 00001000-0009abff : System RAM
>> > >> >> 0009ac00-0009ffff : reserved
>> > >> >> 000c0000-000c7fff : Video ROM
>> > >> >> 000c8000-000c8fff : Adapter ROM
>> > >> >> 000e0000-000fffff : reserved
>> > >> >> 000f0000-000fffff : System ROM
>> > >> >> 00100000-78f36fff : System RAM
>> > >> >> 01000000-01828f14 : Kernel code
>> > >> >> 01828f15-01f3fb3f : Kernel data
>> > >> >> 020bb000-021fdfff : Kernel bss
>> > >> >> 30000000-37ffffff : Crash kernel
>> > >> >> 78f37000-79839fff : reserved
>> > >> >> 79116018-79116018 : APEI ERST
>> > >> >> 7911601c-79116021 : APEI ERST
>> > >> >> 79116028-79116039 : APEI ERST
>> > >> >> 79116040-7911604c : APEI ERST
>> > >> >> 79116050-7911804f : APEI ERST
>> > >> >> 7983a000-79d4afff : ACPI Non-volatile Storage
>> > >> >> 79d4b000-8fffffff : reserved
>> > >> >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> > >> >> c6000000-c70fffff : PCI Bus 0000:2f
>> > >> >> c6000000-c70fffff : PCI Bus 0000:30
>> > >> >> c6000000-c6ffffff : 0000:30:00.0
>> > >> >> c7000000-c701ffff : 0000:30:00.0
>> > >> >> c7100000-c71fffff : PCI Bus 0000:31
>> > >> >> c7100000-c711ffff : 0000:31:00.1
>> > >> >> c7100000-c711ffff : igb
>> > >> >> c7120000-c713ffff : 0000:31:00.0
>> > >> >> c7120000-c713ffff : igb
>> > >> >> c7140000-c7143fff : 0000:31:00.1
>> > >> >> c7140000-c7143fff : igb
>> > >> >> c7144000-c7147fff : 0000:31:00.0
>> > >> >> c7144000-c7147fff : igb
>> > >> >> c7300000-c730ffff : 0000:00:14.0
>> > >> >> c7300000-c730ffff : xhci-hcd
>> > >> >> c7310000-c73107ff : 0000:00:1f.2
>> > >> >> c7310000-c73107ff : ahci
>> > >> >> c7311000-c73113ff : 0000:00:1d.0
>> > >> >> c7311000-c73113ff : ehci_hcd
>> > >> >> c7312000-c73123ff : 0000:00:1a.0
>> > >> >> c7312000-c73123ff : ehci_hcd
>> > >> >> c7314000-c731400f : 0000:00:16.1
>> > >> >> c7315000-c731500f : 0000:00:16.0
>> > >> >> c7316000-c73167ff : 0000:00:11.4
>> > >> >> c7316000-c73167ff : ahci
>> > >> >> c7317000-c7317fff : 0000:00:05.4
>> > >> >> c7ffc000-c7ffcfff : dmar1
>> > >> >> c8000000-c8000000 : <BAD>
>> > >> >> c8000001-c8000001 : <BAD>
>> > >> >> c8000002-c8000002 : <BAD>
>> > >> >> c8000003-c8000003 : <BAD>
>> > >> >
>> > >> > Interesting, what are these from?
>> > >> >
>> > >> >> c8100000-c91fffff : PCI Bus 0000:02
>> > >> >> c8100000-c90fffff : PCI Bus 0000:03
>> > >> >> c8100000-c90fffff : PCI Bus 0000:04
>> > >> >> c8100000-c82fffff : PCI Bus 0000:05
>> > >> >> c8100000-c81fffff : PCI Bus 0000:06
>> > >> >> c8100000-c8100fff : 0000:06:00.0
>> > >> >
>> > >> > You're trying to read the 06:00.0 BAR at 0xc8100000. The windows
>> > >> > above look fine, but the top-level host bridge window is missing.
>> > >> >
>> > >> > The c8100000-c91fffff window is a window through the 00:03.0 Root
>> > >> > Port, which connects bus 00 to bus 02. The 00:03.0 bridge forwards
>> > >> > memory transactions in the c8100000-c91fffff range from its primary
>> > >> > bus (bus 00) to its secondary bus (bus 02).
>> > >> >
>> > >> > But we don't know how those transactions get to bus 00 in the first
>> > >> > place. The host bridge is the connection between the processor and
>> > >> > bus 00, and it should have a window that encloses c8100000-c91fffff.
>> > >> >
>> > >> > Your dmesg log should have information about this bridge, e.g.,
>> > >> > something like this:
>> > >> >
>> > >> > ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
>> > >> > PCI host bridge to bus 0000:00
>> > >> > pci_bus 0000:00: root bus resource [mem 0xc8100000-0xc9ffffff window]
>> > >> >
>> > >> > and there should be a line in /proc/iomem something like this:
>> > >> >
>> > >> > c8100000-c9ffffff : PCI Bus 0000:00
>> > >> >
>> > >> > Bjorn
>> > >
>> > >> 00000000-00000fff : reserved
>> > >> 00001000-0009abff : System RAM
>> > >> 0009ac00-0009ffff : reserved
>> > >> 000c0000-000c7fff : Video ROM
>> > >> 000c8000-000c8fff : Adapter ROM
>> > >> 000e0000-000fffff : reserved
>> > >> 000f0000-000fffff : System ROM
>> > >> 00100000-78f36fff : System RAM
>> > >> 01000000-01828f14 : Kernel code
>> > >> 01828f15-01f3fb3f : Kernel data
>> > >> 020bb000-021fdfff : Kernel bss
>> > >> 30000000-37ffffff : Crash kernel
>> > >> 78f37000-79839fff : reserved
>> > >> 79116018-79116018 : APEI ERST
>> > >> 7911601c-79116021 : APEI ERST
>> > >> 79116028-79116039 : APEI ERST
>> > >> 79116040-7911604c : APEI ERST
>> > >> 79116050-7911804f : APEI ERST
>> > >> 7983a000-79d4afff : ACPI Non-volatile Storage
>> > >> 79d4b000-8fffffff : reserved
>> > >> 80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
>> > >> c6000000-c70fffff : PCI Bus 0000:2f
>> > >> c6000000-c70fffff : PCI Bus 0000:30
>> > >> c6000000-c6ffffff : 0000:30:00.0
>> > >> c7000000-c701ffff : 0000:30:00.0
>> > >> c7100000-c71fffff : PCI Bus 0000:31
>> > >> c7100000-c711ffff : 0000:31:00.1
>> > >> c7100000-c711ffff : igb
>> > >> c7120000-c713ffff : 0000:31:00.0
>> > >> c7120000-c713ffff : igb
>> > >> c7140000-c7143fff : 0000:31:00.1
>> > >> c7140000-c7143fff : igb
>> > >> c7144000-c7147fff : 0000:31:00.0
>> > >> c7144000-c7147fff : igb
>> > >> c7300000-c730ffff : 0000:00:14.0
>> > >> c7300000-c730ffff : xhci-hcd
>> > >> c7310000-c73107ff : 0000:00:1f.2
>> > >> c7310000-c73107ff : ahci
>> > >> c7311000-c73113ff : 0000:00:1d.0
>> > >> c7311000-c73113ff : ehci_hcd
>> > >> c7312000-c73123ff : 0000:00:1a.0
>> > >> c7312000-c73123ff : ehci_hcd
>> > >> c7314000-c731400f : 0000:00:16.1
>> > >> c7315000-c731500f : 0000:00:16.0
>> > >> c7316000-c73167ff : 0000:00:11.4
>> > >> c7316000-c73167ff : ahci
>> > >> c7317000-c7317fff : 0000:00:05.4
>> > >> c7ffc000-c7ffcfff : dmar1
>> > >> c8000000-c8000000 : <BAD>
>> > >> c8000001-c8000001 : <BAD>
>> > >> c8000002-c8000002 : <BAD>
>> > >> c8000003-c8000003 : <BAD>
>> > >> c8100000-c91fffff : PCI Bus 0000:02
>> > >> c8100000-c90fffff : PCI Bus 0000:03
>> > >> c8100000-c90fffff : PCI Bus 0000:04
>> > >> c8100000-c82fffff : PCI Bus 0000:05
>> > >> c8100000-c81fffff : PCI Bus 0000:06
>> > >> c8100000-c8100fff : 0000:06:00.0
>> > >> c8200000-c82fffff : PCI Bus 0000:0a
>> > >> c8200000-c8203fff : 0000:0a:00.0
>> > >> c9100000-c913ffff : 0000:02:00.0
>> > >> fb000000-fb8fffff : PCI Bus 0000:81
>> > >> fb000000-fb8fffff : PCI Bus 0000:82
>> > >> fb000000-fb7fffff : PCI Bus 0000:8c
>> > >> fb000000-fb7fffff : 0000:8c:00.0
>> > >> fb000000-fb7fffff : Plx_Mgr
>> > >> fb800000-fb8fffff : PCI Bus 0000:87
>> > >> fb800000-fb800fff : 0000:87:00.0
>> > >> fb900000-fb900fff : 0000:80:05.4
>> > >> fbffc000-fbffcfff : dmar0
>> > >> fec00000-fecfffff : PNP0003:00
>> > >> fec00000-fec003ff : IOAPIC 0
>> > >> fec01000-fec013ff : IOAPIC 1
>> > >> fec40000-fec403ff : IOAPIC 2
>> > >> fed00000-fed003ff : HPET 0
>> > >> fed00000-fed003ff : PNP0103:00
>> > >> fed12000-fed1200f : pnp 00:01
>> > >> fed12010-fed1201f : pnp 00:01
>> > >> fed1b000-fed1bfff : pnp 00:01
>> > >> fed1c000-fed44fff : reserved
>> > >> fed1c000-fed3ffff : pnp 00:01
>> > >> fed1f410-fed1f414 : iTCO_wdt.0.auto
>> > >> fed45000-fed8bfff : pnp 00:01
>> > >> fee00000-feefffff : pnp 00:01
>> > >> fee00000-fee00fff : Local APIC
>> > >> ff000000-ffffffff : reserved
>> > >> ff000000-ffffffff : pnp 00:01
>> > >> 100000000-47fffffff : System RAM
>> > >> 480000000-4807fffff : PCI Bus 0000:02
>> > >> 480000000-4807fffff : PCI Bus 0000:03
>> > >> 480000000-4807fffff : PCI Bus 0000:04
>> > >> 480000000-4800fffff : PCI Bus 0000:05
>> > >> 480000000-4800fffff : PCI Bus 0000:06
>> > >> 480000000-480003fff : 0000:06:00.0
>> > >> 383fffe00000-383fffefffff : PCI Bus 0000:31
>> > >> 383fffe00000-383fffe1ffff : 0000:31:00.1
>> > >> 383fffe20000-383fffe3ffff : 0000:31:00.1
>> > >> 383fffe40000-383fffe5ffff : 0000:31:00.0
>> > >> 383fffe60000-383fffe7ffff : 0000:31:00.0
>> > >> 383ffff00000-383ffff03fff : 0000:00:04.7
>> > >> 383ffff00000-383ffff03fff : ioatdma
>> > >> 383ffff04000-383ffff07fff : 0000:00:04.6
>> > >> 383ffff04000-383ffff07fff : ioatdma
>> > >> 383ffff08000-383ffff0bfff : 0000:00:04.5
>> > >> 383ffff08000-383ffff0bfff : ioatdma
>> > >> 383ffff0c000-383ffff0ffff : 0000:00:04.4
>> > >> 383ffff0c000-383ffff0ffff : ioatdma
>> > >> 383ffff10000-383ffff13fff : 0000:00:04.3
>> > >> 383ffff10000-383ffff13fff : ioatdma
>> > >> 383ffff14000-383ffff17fff : 0000:00:04.2
>> > >> 383ffff14000-383ffff17fff : ioatdma
>> > >> 383ffff18000-383ffff1bfff : 0000:00:04.1
>> > >> 383ffff18000-383ffff1bfff : ioatdma
>> > >> 383ffff1c000-383ffff1ffff : 0000:00:04.0
>> > >> 383ffff1c000-383ffff1ffff : ioatdma
>> > >> 383ffff21000-383ffff210ff : 0000:00:1f.3
>> > >> 387fffe00000-387fffefffff : PCI Bus 0000:81
>> > >> 387fffe00000-387fffefffff : PCI Bus 0000:82
>> > >> 387fffe00000-387fffefffff : PCI Bus 0000:87
>> > >> 387fffe00000-387fffe03fff : 0000:87:00.0
>> > >> 387ffff00000-387ffff03fff : 0000:80:04.7
>> > >> 387ffff00000-387ffff03fff : ioatdma
>> > >> 387ffff04000-387ffff07fff : 0000:80:04.6
>> > >> 387ffff04000-387ffff07fff : ioatdma
>> > >> 387ffff08000-387ffff0bfff : 0000:80:04.5
>> > >> 387ffff08000-387ffff0bfff : ioatdma
>> > >> 387ffff0c000-387ffff0ffff : 0000:80:04.4
>> > >> 387ffff0c000-387ffff0ffff : ioatdma
>> > >> 387ffff10000-387ffff13fff : 0000:80:04.3
>> > >> 387ffff10000-387ffff13fff : ioatdma
>> > >> 387ffff14000-387ffff17fff : 0000:80:04.2
>> > >> 387ffff14000-387ffff17fff : ioatdma
>> > >> 387ffff18000-387ffff1bfff : 0000:80:04.1
>> > >> 387ffff18000-387ffff1bfff : ioatdma
>> > >> 387ffff1c000-387ffff1ffff : 0000:80:04.0
>> > >> 387ffff1c000-387ffff1ffff : ioatdma
>> > >
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-08-15 21:56 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <CAH=XD5_kqkw=f1g_M69eUHsSDhry32s5VQJXC-_iVa+LZ+vUDA@mail.gmail.com>
[not found] ` <CAH=XD58J+Yf-Rdz_bAJ-e9rU4ipu9opieFd0BuHeYAdX0nmrzg@mail.gmail.com>
2016-08-12 21:14 ` Fwd: PCIe root port programming Kallol Biswas
2016-08-15 17:24 ` Bjorn Helgaas
2016-08-15 18:39 ` Kallol Biswas
2016-08-15 19:06 ` Bjorn Helgaas
2016-08-15 20:43 ` Kallol Biswas
2016-08-15 21:07 ` Bjorn Helgaas
2016-08-15 21:51 ` Kallol Biswas
2016-08-15 21:56 ` Kallol Biswas
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.