* PCI resource allocation not using pref 64-bit
@ 2016-06-22 7:04 Benjamin Herrenschmidt
2016-06-22 15:42 ` Benjamin Herrenschmidt
0 siblings, 1 reply; 2+ messages in thread
From: Benjamin Herrenschmidt @ 2016-06-22 7:04 UTC (permalink / raw)
To: Yinghai Lu; +Cc: Bjorn Helgaas, linux-pci
Hi folks !
So on my POWER9 simulator, I noticed that the PCI code isn't assigning
any BAR to my 64-bit prefetchable window.
This is the trimmed log. Focus on domain 0000, the devices on domain
0001 are 32-bit only. I've also appended an lspci output and a
/proc/iomem output.
I think all that is needed but feel free to ask me if you want more.
You can see that I pass 2 windows for the root bridge, a 32-bit
one and a prefetchable 64-bit one. It's PCIe so anything 64-bit should
be able to hit the prefetchable 64-bit one but that isn't happening.
Note that PCI_REASSIGN_ALL_RSRC is set, the original setup of the
bridges (including the root complex) is to have base > limit, we
expect Linux to fix them up.
Do you have any idea what's going on ? Is that expected ?
Cheers,
Ben.
[ 0.000000] Linux version 4.7.0-rc4-00029-gf55fbbe9-dirty (benh@pasglop) (gcc
version 6.1.1 20160427 (Red Hat Cross 6.1.1-1) (GCC) ) #50 SMP Wed Jun 22 16:18
:18 AEST 2016
[ 0.000000] Initializing IODA3 OPAL PHB /pciex@600c3c0000000
[ 0.000000] PCI host bridge /pciex@600c3c0000000 (primary) ranges:
[ 0.000000] MEM 0x000600c000000000..0x000600c07ffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006000000000000..0x0006001fffffffff -> 0x0006000000000000 (M64 #1..31)
[ 0.000000] Using M64 #31 as default window
[ 0.000000] 256 (255) PE's M32: 0x80000000 [segment=0x800000]
[ 0.000000] M64: 0x2000000000 [segment=0x20000000]
[ 0.000000] Allocated bitmap for 4088 MSIs (base IRQ 0xfe000)
[ 0.000000] Initializing IODA3 OPAL PHB /pciex@600c3c0100000
[ 0.000000] PCI host bridge /pciex@600c3c0100000 ranges:
[ 0.000000] MEM 0x000600c080000000..0x000600c0fffeffff -> 0x0000000080000000
[ 0.000000] MEM 0x0006002000000000..0x0006003fffffffff -> 0x0006002000000000 (M64 #1..15)
[ 0.000000] Using M64 #15 as default window
[ 0.000000] 128 (127) PE's M32: 0x80000000 [segment=0x1000000]
[ 0.000000] M64: 0x2000000000 [segment=0x40000000]
[ 0.000000] Allocated bitmap for 2040 MSIs (base IRQ 0xfd800)
[ 0.000000] Top of RAM: 0x80000000, Total RAM: 0x80000000
.../...
[ 0.030521] PCI: Probing PCI hardware
[ 0.030632] PCI: I/O resource not set for host bridge /pciex@600c3c0000000 (domain 0)
[ 0.031040] PCI host bridge to bus 0000:00
[ 0.031181] pci_bus 0000:00: root bus resource [mem 0x600c000000000-0x600c07ffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.031407] pci_bus 0000:00: root bus resource [mem 0x6000000000000-0x6001fdfffffff 64bit pref]
[ 0.031657] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.031760] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to ff
[ 0.032028] pci 0000:00:00.0: [1014:04c1] type 01 class 0x060400
[ 0.032455] pci 0000:00:00.0: PME# supported from D0 D3hot
[ 0.033466] pci 0000:01:00.0: [1014:033d] type 00 class 0x010400
[ 0.033708] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x0003ffff 64bit]
[ 0.033965] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00ffffff 64bit pref]
[ 0.034227] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x007fffff pref]
[ 0.035008] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 0.035263] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 01
[ 0.035424] PCI: I/O resource not set for host bridge /pciex@600c3c0100000 (domain 1)
[ 0.035887] PCI host bridge to bus 0001:00
[ 0.035980] pci_bus 0001:00: root bus resource [mem 0x600c080000000-0x600c0fffeffff] (bus address [0x80000000-0xfffeffff])
[ 0.036229] pci_bus 0001:00: root bus resource [mem 0x6002000000000-0x6003fbfffffff 64bit pref]
[ 0.036431] pci_bus 0001:00: root bus resource [bus 00-ff]
[ 0.036582] pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to ff
[ 0.036803] pci 0001:00:00.0: [1014:04c1] type 01 class 0x060400
[ 0.037277] pci 0001:00:00.0: PME# supported from D0 D3hot
[ 0.038243] pci 0001:01:00.0: [8086:105e] type 00 class 0x020000
[ 0.038446] pci 0001:01:00.0: reg 0x10: [mem 0x00000000-0x0001ffff]
[ 0.038687] pci 0001:01:00.0: reg 0x14: [mem 0x00000000-0x0003ffff]
[ 0.038880] pci 0001:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff]
[ 0.039150] pci 0001:01:00.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
[ 0.039987] pci 0001:01:00.1: [8086:105e] type 00 class 0x020000
[ 0.040192] pci 0001:01:00.1: reg 0x10: [mem 0x00000000-0x0001ffff]
[ 0.040385] pci 0001:01:00.1: reg 0x14: [mem 0x00000000-0x0003ffff]
[ 0.040626] pci 0001:01:00.1: reg 0x18: [mem 0x00000000-0x0000ffff]
[ 0.040899] pci 0001:01:00.1: reg 0x30: [mem 0x00000000-0x0001ffff pref]
[ 0.041697] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 0.041984] pci_bus 0001:00: busn_res: [bus 00-ff] end is updated to 01
[ 0.042230] pci 0000:00:00.0: BAR 8: assigned [mem 0x600c000000000-0x600c0007fffff]
[ 0.042421] pci 0000:00:00.0: BAR 9: assigned [mem 0x600c000800000-0x600c001ffffff pref]
[ 0.042620] pci 0000:01:00.0: BAR 2: assigned [mem 0x600c001000000-0x600c001ffffff 64bit pref]
[ 0.042900] pci 0000:01:00.0: BAR 6: assigned [mem 0x600c000800000-0x600c000ffffff pref]
[ 0.043095] pci 0000:01:00.0: BAR 0: assigned [mem 0x600c000000000-0x600c00003ffff 64bit]
[ 0.043304] pci 0000:00:00.0: PCI bridge to [bus 01]
[ 0.043465] pci 0000:00:00.0: bridge window [mem 0x600c000000000-0x600c0007fffff]
[ 0.043675] pci 0000:00:00.0: bridge window [mem 0x600c000800000-0x600c001ffffff pref]
[ 0.043925] pci_bus 0000:00: resource 4 [mem 0x600c000000000-0x600c07ffeffff]
[ 0.044112] pci_bus 0000:00: resource 5 [mem 0x6000000000000-0x6001fdfffffff 64bit pref]
[ 0.044300] pci_bus 0000:01: resource 1 [mem 0x600c000000000-0x600c0007fffff]
[ 0.044487] pci_bus 0000:01: resource 2 [mem 0x600c000800000-0x600c001ffffff pref]
[ 0.044698] pci 0001:00:00.0: BAR 8: assigned [mem 0x600c080000000-0x600c080ffffff]
[ 0.044889] pci 0001:00:00.0: BAR 9: assigned [mem 0x600c081000000-0x600c081ffffff pref]
[ 0.045137] pci 0001:01:00.0: BAR 1: assigned [mem 0x600c080000000-0x600c08003ffff]
[ 0.045334] pci 0001:01:00.1: BAR 1: assigned [mem 0x600c080040000-0x600c08007ffff]
[ 0.045531] pci 0001:01:00.0: BAR 0: assigned [mem 0x600c080080000-0x600c08009ffff]
[ 0.045729] pci 0001:01:00.0: BAR 6: assigned [mem 0x600c081000000-0x600c08101ffff pref]
[ 0.045921] pci 0001:01:00.1: BAR 0: assigned [mem 0x600c0800a0000-0x600c0800bffff]
[ 0.046166] pci 0001:01:00.1: BAR 6: assigned [mem 0x600c081020000-0x600c08103ffff pref]
[ 0.046357] pci 0001:01:00.0: BAR 2: assigned [mem 0x600c0800c0000-0x600c0800cffff]
[ 0.046555] pci 0001:01:00.1: BAR 2: assigned [mem 0x600c0800d0000-0x600c0800dffff]
[ 0.046748] pci 0001:00:00.0: PCI bridge to [bus 01]
[ 0.046909] pci 0001:00:00.0: bridge window [mem 0x600c080000000-0x600c080ffffff]
[ 0.047103] pci 0001:00:00.0: bridge window [mem 0x600c081000000-0x600c081ffffff pref]
[ 0.047352] pci_bus 0001:00: resource 4 [mem 0x600c080000000-0x600c0fffeffff]
[ 0.047540] pci_bus 0001:00: resource 5 [mem 0x6002000000000-0x6003fbfffffff 64bit pref]
[ 0.047728] pci_bus 0001:01: resource 1 [mem 0x600c080000000-0x600c080ffffff]
[ 0.047915] pci_bus 0001:01: resource 2 [mem 0x600c081000000-0x600c081ffffff pref]
[ 0.048091] pnv_ioda_reserve_pe: Invalid PE 1536 on PHB#0
[ 0.048240] pci 0000:00 : [PE# 000] Secondary bus 0 associated with PE#0
[ 0.050026] pci 0000:01 : [PE# 001] Secondary bus 1 associated with PE#1
[ 0.051814] pci 0001:00 : [PE# 000] Secondary bus 0 associated with PE#0
[ 0.053600] pci 0001:01 : [PE# 001] Secondary bus 1 associated with PE#1
[ 0.055408] PCI: Domain 0000 has 8 available 32-bit DMA segments
[ 0.055570] pci 0000:01 : [PE# 001] Assign DMA32 space
[ 0.055781] pci 0000:01 : [PE# 001] Setting up 32-bit TCE table at 0..80000000
[ 0.123541] IOMMU table initialized, virtual merging enabled
[ 0.123715] pci 0000:01 : [PE# 001] Setting up window#0 0..7fffffff pg=1000
[ 0.124494] pci 0000:01 : [PE# 001] Enabling 64-bit DMA bypass
[ 0.125467] iommu: Adding device 0000:01:00.0 to group 0
[ 0.125606] PCI: Domain 0001 has 8 available 32-bit DMA segments
[ 0.125720] pci 0001:01 : [PE# 001] Assign DMA32 space
[ 0.125960] pci 0001:01 : [PE# 001] Setting up 32-bit TCE table at 0..80000000
[ 0.183171] pci 0001:01 : [PE# 001] Setting up window#0 0..7fffffff pg=1000
[ 0.183921] pci 0001:01 : [PE# 001] Enabling 64-bit DMA bypass
[ 0.184920] iommu: Adding device 1001:01:00.0 to group 1
[ 0.185079] iommu: Adding device 0001:01:00.1 to group 1
[ 0.188083] EEH: PCI Enhanced I/O Error Handling Enabled
[ 0.188425] PCI: Probing PCI hardware done
# lspci -v
0000:00:00.0 PCI bridge: IBM Device 04c1 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: 80000000-807fffff
Prefetchable memory behind bridge: 80800000-81ffffff
Capabilities: [40] Power Management version 1
Capabilities: [48] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
0000:01:00.0 RAID bus controller: IBM PCI-E IPR SAS Adapter (FPGA)
Subsystem: IBM PCIe2 1.8GB Cache 6Gb SAS RAID Adapter Tri-port (57B5)
Flags: bus master, fast devsel, latency 74, IRQ 128
Memory at 600c000000000 (64-bit, non-prefetchable) [size=256K]
Memory at 600c001000000 (64-bit, prefetchable) [size=16M]
[virtual] Expansion ROM at 600c000800000 [disabled] [size=8M]
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Power Management version 2
Capabilities: [a0] Express Legacy Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
0001:00:00.0 PCI bridge: IBM Device 04c1 (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: 80000000-80ffffff
Prefetchable memory behind bridge: 81000000-81ffffff
Capabilities: [40] Power Management version 1
Capabilities: [48] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [148] #19
0001:01:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Con
troller (rev 06)
Subsystem: IBM Device 0340
Flags: bus master, 66MHz, medium devsel, latency 8, IRQ 26
Memory at 600c080080000 (32-bit, non-prefetchable) [size=128K]
Memory at 600c080000000 (32-bit, non-prefetchable) [size=256K]
Memory at 600c0800c0000 (32-bit, non-prefetchable) [size=64K]
[virtual] Expansion ROM at 600c081000000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 0
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: e1000e
0001:01:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Con
troller (rev 06)
Subsystem: IBM Device 0340
Flags: bus master, 66MHz, medium devsel, latency 8, IRQ 27
Memory at 600c0800a0000 (32-bit, non-prefetchable) [size=128K]
Memory at 600c080040000 (32-bit, non-prefetchable) [size=256K]
Memory at 600c0800d0000 (32-bit, non-prefetchable) [size=64K]
[virtual] Expansion ROM at 600c081020000 [disabled] [size=128K]
Capabilities: [c8] Power Management version 0
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [e0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: e1000e
# cat /proc/iomem
00000000-7fffffff : System RAM
6000000000000-6001fdfffffff : /pciex@600c3c0000000
6002000000000-6003fbfffffff : /pciex@600c3c0100000
600c000000000-600c07ffeffff : /pciex@600c3c0000000
600c000000000-600c0007fffff : PCI Bus 0000:01
600c000000000-600c00003ffff : 0000:01:00.0
600c000800000-600c001ffffff : PCI Bus 0000:01
600c000800000-600c000ffffff : 0000:01:00.0
600c001000000-600c001ffffff : 0000:01:00.0
600c080000000-600c0fffeffff : /pciex@600c3c0100000
600c080000000-600c080ffffff : PCI Bus 0001:01
600c080000000-600c08003ffff : 1001:01:00.0
600c080000000-600c08003ffff : e1000e
600c080040000-600c08007ffff : 0001:01:00.1
600c080040000-600c08007ffff : e1000e
600c080080000-600c08009ffff : 1001:01:00.0
600c080080000-600c08009ffff : e1000e
600c0800a0000-600c0800bffff : 0001:01:00.1
600c0800a0000-600c0800bffff : e1000e
600c0800c0000-600c0800cffff : 1001:01:00.0
600c0800c0000-600c0800cffff : e1000e
600c0800d0000-600c0800dffff : 0001:01:00.1
600c0800d0000-600c0800dffff : e1000e
600c081000000-600c081ffffff : PCI Bus 0001:01
600c081000000-600c08101ffff : 1001:01:00.0
600c081020000-600c08103ffff : 0001:01:00.1
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: PCI resource allocation not using pref 64-bit
2016-06-22 7:04 PCI resource allocation not using pref 64-bit Benjamin Herrenschmidt
@ 2016-06-22 15:42 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 2+ messages in thread
From: Benjamin Herrenschmidt @ 2016-06-22 15:42 UTC (permalink / raw)
To: Yinghai Lu; +Cc: Bjorn Helgaas, linux-pci
On Wed, 2016-06-22 at 17:04 +1000, Benjamin Herrenschmidt wrote:
> Hi folks !
>
> So on my POWER9 simulator, I noticed that the PCI code isn't assigning
> any BAR to my 64-bit prefetchable window.
Problem found ... for various reasons the "windows" registers of the
root complex are emulated in firmware and the emulation wasn't properly
masking (and OR'ing 1) in the prefetchable ones, thus Linux was ditching
them.
Sorry for the noise..
Cheers,
Ben.
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2016-06-22 15:42 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-22 7:04 PCI resource allocation not using pref 64-bit Benjamin Herrenschmidt
2016-06-22 15:42 ` Benjamin Herrenschmidt
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.