[PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Phil Edworthy]

Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
the board's dts, even if we aren't using PCIe.

Signed-off-by: Phil Edworthy <phil.edworthy@renesas.com>
---
 arch/arm/boot/dts/r8a7791-henninger.dts | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm/boot/dts/r8a7791-henninger.dts b/arch/arm/boot/dts/r8a7791-henninger.dts
index cc6d992..6e5d779 100644
--- a/arch/arm/boot/dts/r8a7791-henninger.dts
+++ b/arch/arm/boot/dts/r8a7791-henninger.dts
@@ -78,6 +78,16 @@
 		states = <3300000 1
 			  1800000 0>;
 	};
+
+	clocks {
+		/* External PCIe bus clock - not used */
+		pcie_bus_clk: pcie_bus_clk {
+			compatible = "fixed-clock";
+			#clock-cells = <0>;
+			clock-frequency = <0>;
+			clock-output-names = "pcie_bus";
+		};
+	};
 };
 
 &extal_clk {
-- 
1.9.3

Re: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Sergei Shtylyov]

Hello.

On 05/28/2014 11:24 AM, Phil Edworthy wrote:

> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
> the board's dts, even if we aren't using PCIe.

    I don't quite understand that last part: there's PCIe slot in the 
Henninger's schematics (although it doesn't seem to be soldered in the early 
board version).

WBR, Sergei

Re: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Sergei Shtylyov]

Hello.

On 06/12/2014 01:03 AM, Sergei Shtylyov wrote:

>> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
>> the board's dts, even if we aren't using PCIe.

>     I don't quite understand that last part: there's PCIe slot in the
> Henninger's schematics (although it doesn't seem to be soldered in the early
> board version).

    In fact, we're in process of verifying PCIe on Henninger (soldered the 
connector); at least the configuration space accesses work OK...

WBR, Sergei

RE: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Phil Edworthy]

Hi Sergei,

On 12 June 2014 00:25, Sergei wrote:
> On 06/12/2014 01:03 AM, Sergei Shtylyov wrote:
> 
> >> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
> >> the board's dts, even if we aren't using PCIe.
> 
> >     I don't quite understand that last part: there's PCIe slot in the
> > Henninger's schematics (although it doesn't seem to be soldered in the
> early
> > board version).
> 
>     In fact, we're in process of verifying PCIe on Henninger (soldered the
> connector); at least the configuration space accesses work OK...

At the time of writing those patches, I had no idea if the Henninger board had a PCIe slot or not.

Also, please see v2 of the patches. You no longer need to specify the PCIe bus clock if it's always on.

Regards
Phil

RE: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Phil Edworthy]

Hi Sergei,

On 12 June 2014 00:25, Sergei wrote:
> On 06/12/2014 01:03 AM, Sergei Shtylyov wrote:
> 
> >> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
> >> the board's dts, even if we aren't using PCIe.
> 
> >     I don't quite understand that last part: there's PCIe slot in the
> > Henninger's schematics (although it doesn't seem to be soldered in the
> early
> > board version).
> 
>     In fact, we're in process of verifying PCIe on Henninger (soldered the
> connector); at least the configuration space accesses work OK...

Do you mean the internal Root-Complex config accesses work ok, or card ones?

One thing we noticed on the Koelsch board is that some cards need Aux 3.3V to be supplied before they will work. By default on the Koelsch board, this is not supplied - you have to add link R719.

Regards
Phil

Re: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Sergei Shtylyov]

Hello.

On 06/12/2014 12:36 PM, Phil Edworthy wrote:

>>>> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
>>>> the board's dts, even if we aren't using PCIe.

>>>      I don't quite understand that last part: there's PCIe slot in the
>>> Henninger's schematics (although it doesn't seem to be soldered in the
>>> early board version).

>>      In fact, we're in process of verifying PCIe on Henninger (soldered the
>> connector); at least the configuration space accesses work OK...

> At the time of writing those patches, I had no idea if the Henninger board had a PCIe slot or not.

    Ah, that explains it...

> Also, please see v2 of the patches. You no longer need to specify the PCIe bus clock if it's always on.

    I know. But we still need to enable PCIe bridge on Henninger.

> Regards
> Phil

WBR, Sergei

Re: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Sergei Shtylyov]

Hello.

On 06/12/2014 07:41 PM, Phil Edworthy wrote:

>>>> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
>>>> the board's dts, even if we aren't using PCIe.

>>>      I don't quite understand that last part: there's PCIe slot in the
>>> Henninger's schematics (although it doesn't seem to be soldered in the
>>> early board version).

>>      In fact, we're in process of verifying PCIe on Henninger (soldered the
>> connector); at least the configuration space accesses work OK...

> Do you mean the internal Root-Complex config accesses work ok, or card ones?

    We inserted an Intel Wi-Fi card and saw PCI-PCI bridge and network 
controller in lspci's output. Another (Ethernet) card didn't work though...

> One thing we noticed on the Koelsch board is that some cards need Aux 3.3V to be supplied before they will work. By default on the Koelsch board, this is not supplied - you have to add link R719.

    Thanks for the info.

> Regards
> Phil

WBR, Sergei

Re: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Sergei Shtylyov]

Hello.

On 06/12/2014 08:47 PM, Sergei Shtylyov wrote:

>>>>> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
>>>>> the board's dts, even if we aren't using PCIe.

>>>>      I don't quite understand that last part: there's PCIe slot in the
>>>> Henninger's schematics (although it doesn't seem to be soldered in the
>>>> early board version).

>>>      In fact, we're in process of verifying PCIe on Henninger (soldered the
>>> connector); at least the configuration space accesses work OK...

>> Do you mean the internal Root-Complex config accesses work ok, or card ones?

>     We inserted an Intel Wi-Fi card and saw PCI-PCI bridge and network
> controller in lspci's output. Another (Ethernet) card didn't work though...

    That PCI-PCI bridge has ID 1912:001f which corresponds to Renesas. I guess 
it's a part of the PCIe controller and then I wonder why this device is not 
seen when no card is inserted...

WBR, Sergei

RE: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Phil Edworthy]

Hi Sergei,

On 16 June 2014 23:55, Sergei wrote:
> On 06/12/2014 08:47 PM, Sergei Shtylyov wrote:
> 
> >>>>> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
> >>>>> the board's dts, even if we aren't using PCIe.
> 
> >>>>      I don't quite understand that last part: there's PCIe slot in the
> >>>> Henninger's schematics (although it doesn't seem to be soldered in the
> >>>> early board version).
> 
> >>>      In fact, we're in process of verifying PCIe on Henninger (soldered the
> >>> connector); at least the configuration space accesses work OK...
> 
> >> Do you mean the internal Root-Complex config accesses work ok, or card
> ones?
> 
> >     We inserted an Intel Wi-Fi card and saw PCI-PCI bridge and network
> > controller in lspci's output. Another (Ethernet) card didn't work though...
> 
>     That PCI-PCI bridge has ID 1912:001f which corresponds to Renesas. I guess
> it's a part of the PCIe controller and then I wonder why this device is not
> seen when no card is inserted...

Simply because the hardware does not support hot plug. So, during probe the driver checks for link up, and if so it will call pci_common_init_dev().

Regards
Phil

Re: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Sergei Shtylyov]

On 06/17/2014 11:41 AM, Phil Edworthy wrote:

>>>>>>> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
>>>>>>> the board's dts, even if we aren't using PCIe.

>>>>>>       I don't quite understand that last part: there's PCIe slot in the
>>>>>> Henninger's schematics (although it doesn't seem to be soldered in the
>>>>>> early board version).

>>>>>    In fact, we're in process of verifying PCIe on Henninger (soldered the
>>>>> connector); at least the configuration space accesses work OK...

>>>> Do you mean the internal Root-Complex config accesses work ok, or card
>>>> ones?

>>>      We inserted an Intel Wi-Fi card and saw PCI-PCI bridge and network
>>> controller in lspci's output. Another (Ethernet) card didn't work though...

    We have found firmware for the Intel Centrino card (it only has BARs in 
the memory space), it loaded successfully but WLAN interface failed to come up 
anyway:

root@10.0.0.111:~# ifconfig wlan0 11.0.0.2
iwlwifi 0000:01:00.0: L1 Disabled; Enabling L0S
iwlwifi 0000:01:00.0: Radio type=0x2-0x1-0x0
iwlwifi 0000:01:00.0: Failed to load firmware chunk!
iwlwifi 0000:01:00.0: Could not load the [0] uCode section
iwlwifi 0000:01:00.0: Failed to run INIT ucode: -110
iwlwifi 0000:01:00.0: Unable to initialize device.
SIOCSIFFLAGS: Connection timed out

    So I wanted to ask: how did you test PCIe?

>>      That PCI-PCI bridge has ID 1912:001f which corresponds to Renesas. I guess
>> it's a part of the PCIe controller and then I wonder why this device is not
>> seen when no card is inserted...

> Simply because the hardware does not support hot plug. So, during probe the driver checks for link up, and if so it will call pci_common_init_dev().

    Not sure how it's connected, could you elaborate?
    And you don't indicate probe error when there's no link anyway -- which is 
strange.
    As I already said, the PCI-PCI bridge's space doesn't look correct:

root@10.0.0.111:~# lspci -v
00:00.0 PCI bridge: Unknown device 1912:001f (prog-if 00 [Normal decode])
         Flags: bus master, fast devsel, latency 0
         Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
         I/O behind bridge: 00000000-00000fff
         Memory behind bridge: 00000000-000fffff
         Prefetchable memory behind bridge: 00000000-000fffff
         Capabilities: [40] Power Management version 3
         Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/0
Enable+
         Capabilities: [70] Express Root Port (Slot-) IRQ 0
         Capabilities: [100] Virtual Channel
         Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00

01:00.0 Network controller: Intel Corporation Unknown device 088e (rev 24)
         Subsystem: Intel Corporation Unknown device 4060
         Flags: bus master, fast devsel, latency 0, IRQ 418
         Memory at fe200000 (64-bit, non-prefetchable) [size=8K]
         Capabilities: [c8] Power Management version 3
         Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
Enable+
         Capabilities: [e0] Express Endpoint IRQ 0
         Capabilities: [100] Advanced Error Reporting
         Capabilities: [140] Device Serial Number cf-9c-3e-ff-ff-87-d9-c4

    The memory behind bridge should include the 0xfe2000000 address in card's 
BAR0 but it doesn't... I/O and prefetchable memory behind bridge also don't 
look right. The strange thing is the kernel reassigns the memory behind 
bridge's range but never writes it back into the PCI-PCI bridge:

pcieport 0000:00:00.0: BAR 8: assigned [mem 0xfe200000-0xfe2fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0xfe200000-0xfe201fff 64bit]

> Regards
> Phil

WBR, Sergei

RE: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Phil Edworthy]

Hi Sergei,

On 19 June 2014 23:02, Sergei wrote:
> On 06/17/2014 11:41 AM, Phil Edworthy wrote:
> 
> >>>>>>> Since the CPU's dtsi refers to pcie_bus_clock, we need an entry in
> >>>>>>> the board's dts, even if we aren't using PCIe.
> 
> >>>>>>       I don't quite understand that last part: there's PCIe slot in the
> >>>>>> Henninger's schematics (although it doesn't seem to be soldered in
> the
> >>>>>> early board version).
> 
> >>>>>    In fact, we're in process of verifying PCIe on Henninger (soldered
> the
> >>>>> connector); at least the configuration space accesses work OK...
> 
> >>>> Do you mean the internal Root-Complex config accesses work ok, or
> card
> >>>> ones?
> 
> >>>      We inserted an Intel Wi-Fi card and saw PCI-PCI bridge and network
> >>> controller in lspci's output. Another (Ethernet) card didn't work though...
> 
>     We have found firmware for the Intel Centrino card (it only has BARs in
> the memory space), it loaded successfully but WLAN interface failed to come
> up
> anyway:
> 
> root@10.0.0.111:~# ifconfig wlan0 11.0.0.2
> iwlwifi 0000:01:00.0: L1 Disabled; Enabling L0S
> iwlwifi 0000:01:00.0: Radio type=0x2-0x1-0x0
> iwlwifi 0000:01:00.0: Failed to load firmware chunk!
> iwlwifi 0000:01:00.0: Could not load the [0] uCode section
> iwlwifi 0000:01:00.0: Failed to run INIT ucode: -110
> iwlwifi 0000:01:00.0: Unable to initialize device.
> SIOCSIFFLAGS: Connection timed out
> 
>     So I wanted to ask: how did you test PCIe?

I used the following cards:
  Intel PRO/1000 PT Server Adapter
  Intel Gigabit CT Desktop Adapter
  Intel Ethernet Server Adapter I350-T2
  TP-LINK TG-3468 (r8618 based)
I also used three cards on a PCIe switch (PEX8604). The PT Server card support MSI, but not MSI-X, the other Intel cards support MSI-X, and used "pci=nomsi" cmd line arg to check INTx pin swizzling.

However, I could not find a card that used I/O, all use memory resources.


> >>      That PCI-PCI bridge has ID 1912:001f which corresponds to Renesas. I
> guess
> >> it's a part of the PCIe controller and then I wonder why this device is not
> >> seen when no card is inserted...
> 
> > Simply because the hardware does not support hot plug. So, during probe
> the driver checks for link up, and if so it will call pci_common_init_dev().
> 
>     Not sure how it's connected, could you elaborate?
I'm not sure what you are asking. The Koelsch board has PCIe connected, no mods required (except adding a link to provide the 3.3v aux supply for one card).

>     And you don't indicate probe error when there's no link anyway -- which is
> strange.
If there is no link up during probe, it will not be able to get a link up later on, as the R-Car device does not support PCIe hot plug. If link up fails, then the driver will report this, e.g.:
rcar-pcie fe000000.pcie: PCI: PCIe link down


>     As I already said, the PCI-PCI bridge's space doesn't look correct:
> 
> root@10.0.0.111:~# lspci -v
> 00:00.0 PCI bridge: Unknown device 1912:001f (prog-if 00 [Normal decode])
>          Flags: bus master, fast devsel, latency 0
>          Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
>          I/O behind bridge: 00000000-00000fff
>          Memory behind bridge: 00000000-000fffff
>          Prefetchable memory behind bridge: 00000000-000fffff
>          Capabilities: [40] Power Management version 3
>          Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/0
> Enable+
>          Capabilities: [70] Express Root Port (Slot-) IRQ 0
>          Capabilities: [100] Virtual Channel
>          Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00
> 
> 01:00.0 Network controller: Intel Corporation Unknown device 088e (rev 24)
>          Subsystem: Intel Corporation Unknown device 4060
>          Flags: bus master, fast devsel, latency 0, IRQ 418
>          Memory at fe200000 (64-bit, non-prefetchable) [size=8K]
>          Capabilities: [c8] Power Management version 3
>          Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
> Enable+
>          Capabilities: [e0] Express Endpoint IRQ 0
>          Capabilities: [100] Advanced Error Reporting
>          Capabilities: [140] Device Serial Number cf-9c-3e-ff-ff-87-d9-c4
> 
>     The memory behind bridge should include the 0xfe2000000 address in
> card's
> BAR0 but it doesn't... I/O and prefetchable memory behind bridge also don't
> look right. The strange thing is the kernel reassigns the memory behind
> bridge's range but never writes it back into the PCI-PCI bridge:
> 
> pcieport 0000:00:00.0: BAR 8: assigned [mem 0xfe200000-0xfe2fffff]
> pci 0000:01:00.0: BAR 0: assigned [mem 0xfe200000-0xfe201fff 64bit]

That does look very odd. This is what I get:
root@koelsch:~# lspci -v
00:00.0 PCI bridge: Renesas Technology Corp. Device 001f (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: fe200000-fe2fffff
        Prefetchable memory behind bridge: 38000000-380fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00

01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: Intel Corporation Gigabit CT Desktop Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 148
        Memory at fe280000 (32-bit, non-prefetchable) [size\x128K]
        Memory at fe200000 (32-bit, non-prefetchable) [sizeQ2K]
        I/O ports at 1000 [disabled] [size2]
        Memory at fe2a0000 (32-bit, non-prefetchable) [size\x16K]
        [virtual] Expansion ROM at 38000000 [disabled] [size%6K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-19-9d-31
        Kernel driver in use: e1000e

Regards
Phil

Re: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Sergei Shtylyov]

Hello.

On 06/23/2014 01:19 PM, Phil Edworthy wrote:

[...]

>>      So I wanted to ask: how did you test PCIe?

> I used the following cards:
>    Intel PRO/1000 PT Server Adapter
>    Intel Gigabit CT Desktop Adapter
>    Intel Ethernet Server Adapter I350-T2
>    TP-LINK TG-3468 (r8618 based)

    My boss (helping me with the remote debugging on site) said we had tried 
all these cards on Henninger and the Intel cards just didn't get detected. The 
TP-LINK card required a firmware which we're to find yet...

> However, I could not find a card that used I/O, all use memory resources.

    The Intel cards seem to have I/O resources, it's probably the driver that 
prefers to always use MMIO...

>>>>       That PCI-PCI bridge has ID 1912:001f which corresponds to Renesas. I
>>>> guess
>>>> it's a part of the PCIe controller and then I wonder why this device is not
>>>> seen when no card is inserted...

>>> Simply because the hardware does not support hot plug. So, during probe
>> the driver checks for link up, and if so it will call pci_common_init_dev().

>>      Not sure how it's connected, could you elaborate?

> I'm not sure what you are asking. The Koelsch board has PCIe connected, no mods required (except adding a link to provide the 3.3v aux supply for one card).

    Actually I meant to ask how the lack of hot plug and the visibility of the 
built-in bridge is connected. Now I've seen that kernel dies scanning bus with 
no PCIe link for some (yet unobvious) reason...

>>      And you don't indicate probe error when there's no link anyway -- which is
>> strange.

> If there is no link up during probe, it will not be able to get a link up later on, as the R-Car device does not support PCIe hot plug. If link up fails, then the driver will report this, e.g.:
> rcar-pcie fe000000.pcie: PCI: PCIe link down

    That I saw perfectly well. :-) I was not asking about the messages. Well, 
we're discussing this matter in another thread already...

>>      As I already said, the PCI-PCI bridge's space doesn't look correct:

>> root@10.0.0.111:~# lspci -v
>> 00:00.0 PCI bridge: Unknown device 1912:001f (prog-if 00 [Normal decode])
>>           Flags: bus master, fast devsel, latency 0
>>           Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
>>           I/O behind bridge: 00000000-00000fff
>>           Memory behind bridge: 00000000-000fffff
>>           Prefetchable memory behind bridge: 00000000-000fffff
>>           Capabilities: [40] Power Management version 3
>>           Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ Queue=0/0
>> Enable+
>>           Capabilities: [70] Express Root Port (Slot-) IRQ 0
>>           Capabilities: [100] Virtual Channel
>>           Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00
>>
>> 01:00.0 Network controller: Intel Corporation Unknown device 088e (rev 24)
>>           Subsystem: Intel Corporation Unknown device 4060
>>           Flags: bus master, fast devsel, latency 0, IRQ 418
>>           Memory at fe200000 (64-bit, non-prefetchable) [size=8K]
>>           Capabilities: [c8] Power Management version 3
>>           Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
>> Enable+
>>           Capabilities: [e0] Express Endpoint IRQ 0
>>           Capabilities: [100] Advanced Error Reporting
>>           Capabilities: [140] Device Serial Number cf-9c-3e-ff-ff-87-d9-c4

>>      The memory behind bridge should include the 0xfe2000000 address in
>> card's
>> BAR0 but it doesn't... I/O and prefetchable memory behind bridge also don't
>> look right. The strange thing is the kernel reassigns the memory behind
>> bridge's range but never writes it back into the PCI-PCI bridge:

>> pcieport 0000:00:00.0: BAR 8: assigned [mem 0xfe200000-0xfe2fffff]
>> pci 0000:01:00.0: BAR 0: assigned [mem 0xfe200000-0xfe201fff 64bit]

> That does look very odd. This is what I get:
> root@koelsch:~# lspci -v
> 00:00.0 PCI bridge: Renesas Technology Corp. Device 001f (prog-if 00 [Normal decode])
>          Flags: bus master, fast devsel, latency 0
>          Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
>          I/O behind bridge: 00001000-00001fff
>          Memory behind bridge: fe200000-fe2fffff
>          Prefetchable memory behind bridge: 38000000-380fffff
>          Capabilities: [40] Power Management version 3
>          Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>          Capabilities: [70] Express Root Port (Slot-), MSI 00
>          Capabilities: [100] Virtual Channel
>          Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00
>
> 01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
>          Subsystem: Intel Corporation Gigabit CT Desktop Adapter
>          Flags: bus master, fast devsel, latency 0, IRQ 148
>          Memory at fe280000 (32-bit, non-prefetchable) [size\x128K]
>          Memory at fe200000 (32-bit, non-prefetchable) [sizeQ2K]
>          I/O ports at 1000 [disabled] [size2]
>          Memory at fe2a0000 (32-bit, non-prefetchable) [size\x16K]
>          [virtual] Expansion ROM at 38000000 [disabled] [size%6K]
>          Capabilities: [c8] Power Management version 2
>          Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>          Capabilities: [e0] Express Endpoint, MSI 00
>          Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>          Capabilities: [100] Advanced Error Reporting
>          Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-19-9d-31
>          Kernel driver in use: e1000e

    Yes, the bridge looks to be set up correctly here. I've tried to test PCIe 
on the Koelsch board (without the 3.3VAUX resistor soldered though) and either 
a PCIe link didn't get detected or the kernel died horrible death.
Trying to bypass the link check in the driver also resulted in a horrible 
kernel death as described in another thread...

> Regards
> Phil

WBR, Sergei

RE: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Phil Edworthy]

Hi Sergei,

On 24 June 2014 22:47, Sergei wrote:
> On 06/23/2014 01:19 PM, Phil Edworthy wrote:
> 
> [...]
> 
>>>      So I wanted to ask: how did you test PCIe?
> 
>> I used the following cards:
>>    Intel PRO/1000 PT Server Adapter
>>    Intel Gigabit CT Desktop Adapter
>>    Intel Ethernet Server Adapter I350-T2
>>    TP-LINK TG-3468 (r8618 based)
> 
>     My boss (helping me with the remote debugging on site) said we had tried
> all these cards on Henninger and the Intel cards just didn't get detected. The
> TP-LINK card required a firmware which we're to find yet...
Henninger is a new board and PCIe has not been proven on this yet, right? From
the output below, it sounds like you have some HW issues.

BTW, for the TP-LINK card, I used a driver from http://code.google.com/p/r8168
Although there is an in-kernel driver that loads for this card, I had some
issues with it (can't remember what).

>> However, I could not find a card that used I/O, all use memory resources.
> 
>     The Intel cards seem to have I/O resources, it's probably the driver that
> prefers to always use MMIO...
That's true... though I would prefer to not modify the drivers when trying to
get it working.

>>>>>       That PCI-PCI bridge has ID 1912:001f which corresponds to Renesas. I
>>>>> guess
>>>>> it's a part of the PCIe controller and then I wonder why this device is
> not
>>>>> seen when no card is inserted...
> 
>>>> Simply because the hardware does not support hot plug. So, during
> probe
>>> the driver checks for link up, and if so it will call pci_common_init_dev().
> 
>>>      Not sure how it's connected, could you elaborate?
> 
>> I'm not sure what you are asking. The Koelsch board has PCIe connected,
> no mods required (except adding a link to provide the 3.3v aux supply for
> one card).
> 
>     Actually I meant to ask how the lack of hot plug and the visibility of the
> built-in bridge is connected. Now I've seen that kernel dies scanning bus with
> no PCIe link for some (yet unobvious) reason...
Right, just that if hot plug isn't suppoted and there is no link up during
probe, I don't think there is any point trying later on.

>>>      And you don't indicate probe error when there's no link anyway --
> which is
>>> strange.
> 
>> If there is no link up during probe, it will not be able to get a link up later on,
> as the R-Car device does not support PCIe hot plug. If link up fails, then the
> driver will report this, e.g.:
>> rcar-pcie fe000000.pcie: PCI: PCIe link down
> 
>     That I saw perfectly well. :-) I was not asking about the messages. Well,
> we're discussing this matter in another thread already...
> 
>>>      As I already said, the PCI-PCI bridge's space doesn't look correct:
> 
>>> root@10.0.0.111:~# lspci -v
>>> 00:00.0 PCI bridge: Unknown device 1912:001f (prog-if 00 [Normal
> decode])
>>>           Flags: bus master, fast devsel, latency 0
>>>           Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
>>>           I/O behind bridge: 00000000-00000fff
>>>           Memory behind bridge: 00000000-000fffff
>>>           Prefetchable memory behind bridge: 00000000-000fffff
>>>           Capabilities: [40] Power Management version 3
>>>           Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
> Queue=0/0
>>> Enable+
>>>           Capabilities: [70] Express Root Port (Slot-) IRQ 0
>>>           Capabilities: [100] Virtual Channel
>>>           Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00
>>>
>>> 01:00.0 Network controller: Intel Corporation Unknown device 088e (rev
> 24)
>>>           Subsystem: Intel Corporation Unknown device 4060
>>>           Flags: bus master, fast devsel, latency 0, IRQ 418
>>>           Memory at fe200000 (64-bit, non-prefetchable) [size=8K]
>>>           Capabilities: [c8] Power Management version 3
>>>           Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+
> Queue=0/0
>>> Enable+
>>>           Capabilities: [e0] Express Endpoint IRQ 0
>>>           Capabilities: [100] Advanced Error Reporting
>>>           Capabilities: [140] Device Serial Number cf-9c-3e-ff-ff-87-d9-c4
> 
>>>      The memory behind bridge should include the 0xfe2000000 address in
>>> card's
>>> BAR0 but it doesn't... I/O and prefetchable memory behind bridge also
> don't
>>> look right. The strange thing is the kernel reassigns the memory behind
>>> bridge's range but never writes it back into the PCI-PCI bridge:
> 
>>> pcieport 0000:00:00.0: BAR 8: assigned [mem 0xfe200000-0xfe2fffff]
>>> pci 0000:01:00.0: BAR 0: assigned [mem 0xfe200000-0xfe201fff 64bit]
> 
>> That does look very odd. This is what I get:
>> root@koelsch:~# lspci -v
>> 00:00.0 PCI bridge: Renesas Technology Corp. Device 001f (prog-if 00
> [Normal decode])
>>          Flags: bus master, fast devsel, latency 0
>>          Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
>>          I/O behind bridge: 00001000-00001fff
>>          Memory behind bridge: fe200000-fe2fffff
>>          Prefetchable memory behind bridge: 38000000-380fffff
>>          Capabilities: [40] Power Management version 3
>>          Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>>          Capabilities: [70] Express Root Port (Slot-), MSI 00
>>          Capabilities: [100] Virtual Channel
>>          Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00
>>
>> 01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
> Connection
>>          Subsystem: Intel Corporation Gigabit CT Desktop Adapter
>>          Flags: bus master, fast devsel, latency 0, IRQ 148
>>          Memory at fe280000 (32-bit, non-prefetchable) [size\x128K]
>>          Memory at fe200000 (32-bit, non-prefetchable) [sizeQ2K]
>>          I/O ports at 1000 [disabled] [size2]
>>          Memory at fe2a0000 (32-bit, non-prefetchable) [size\x16K]
>>          [virtual] Expansion ROM at 38000000 [disabled] [size%6K]
>>          Capabilities: [c8] Power Management version 2
>>          Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>>          Capabilities: [e0] Express Endpoint, MSI 00
>>          Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>>          Capabilities: [100] Advanced Error Reporting
>>          Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-19-9d-31
>>          Kernel driver in use: e1000e
> 
>     Yes, the bridge looks to be set up correctly here. I've tried to test PCIe
> on the Koelsch board (without the 3.3VAUX resistor soldered though) and
> either
> a PCIe link didn't get detected or the kernel died horrible death.
> Trying to bypass the link check in the driver also resulted in a horrible
> kernel death as described in another thread...
Ok, I'll have a look at bypassing the link check to see what happens.

Cheers
Phil