RE: [PATCH 8/8] ARM: shmobile: henninger: Add dummy PCIe bus clock

[Phil Edworthy <phil.edworthy@renesas.com>]

Hi Sergei,

On 24 June 2014 22:47, Sergei wrote:
> On 06/23/2014 01:19 PM, Phil Edworthy wrote:
> 
> [...]
> 
>>>      So I wanted to ask: how did you test PCIe?
> 
>> I used the following cards:
>>    Intel PRO/1000 PT Server Adapter
>>    Intel Gigabit CT Desktop Adapter
>>    Intel Ethernet Server Adapter I350-T2
>>    TP-LINK TG-3468 (r8618 based)
> 
>     My boss (helping me with the remote debugging on site) said we had tried
> all these cards on Henninger and the Intel cards just didn't get detected. The
> TP-LINK card required a firmware which we're to find yet...
Henninger is a new board and PCIe has not been proven on this yet, right? From
the output below, it sounds like you have some HW issues.

BTW, for the TP-LINK card, I used a driver from http://code.google.com/p/r8168
Although there is an in-kernel driver that loads for this card, I had some
issues with it (can't remember what).

>> However, I could not find a card that used I/O, all use memory resources.
> 
>     The Intel cards seem to have I/O resources, it's probably the driver that
> prefers to always use MMIO...
That's true... though I would prefer to not modify the drivers when trying to
get it working.

>>>>>       That PCI-PCI bridge has ID 1912:001f which corresponds to Renesas. I
>>>>> guess
>>>>> it's a part of the PCIe controller and then I wonder why this device is
> not
>>>>> seen when no card is inserted...
> 
>>>> Simply because the hardware does not support hot plug. So, during
> probe
>>> the driver checks for link up, and if so it will call pci_common_init_dev().
> 
>>>      Not sure how it's connected, could you elaborate?
> 
>> I'm not sure what you are asking. The Koelsch board has PCIe connected,
> no mods required (except adding a link to provide the 3.3v aux supply for
> one card).
> 
>     Actually I meant to ask how the lack of hot plug and the visibility of the
> built-in bridge is connected. Now I've seen that kernel dies scanning bus with
> no PCIe link for some (yet unobvious) reason...
Right, just that if hot plug isn't suppoted and there is no link up during
probe, I don't think there is any point trying later on.

>>>      And you don't indicate probe error when there's no link anyway --
> which is
>>> strange.
> 
>> If there is no link up during probe, it will not be able to get a link up later on,
> as the R-Car device does not support PCIe hot plug. If link up fails, then the
> driver will report this, e.g.:
>> rcar-pcie fe000000.pcie: PCI: PCIe link down
> 
>     That I saw perfectly well. :-) I was not asking about the messages. Well,
> we're discussing this matter in another thread already...
> 
>>>      As I already said, the PCI-PCI bridge's space doesn't look correct:
> 
>>> root@10.0.0.111:~# lspci -v
>>> 00:00.0 PCI bridge: Unknown device 1912:001f (prog-if 00 [Normal
> decode])
>>>           Flags: bus master, fast devsel, latency 0
>>>           Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
>>>           I/O behind bridge: 00000000-00000fff
>>>           Memory behind bridge: 00000000-000fffff
>>>           Prefetchable memory behind bridge: 00000000-000fffff
>>>           Capabilities: [40] Power Management version 3
>>>           Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
> Queue=0/0
>>> Enable+
>>>           Capabilities: [70] Express Root Port (Slot-) IRQ 0
>>>           Capabilities: [100] Virtual Channel
>>>           Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00
>>>
>>> 01:00.0 Network controller: Intel Corporation Unknown device 088e (rev
> 24)
>>>           Subsystem: Intel Corporation Unknown device 4060
>>>           Flags: bus master, fast devsel, latency 0, IRQ 418
>>>           Memory at fe200000 (64-bit, non-prefetchable) [size=8K]
>>>           Capabilities: [c8] Power Management version 3
>>>           Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+
> Queue=0/0
>>> Enable+
>>>           Capabilities: [e0] Express Endpoint IRQ 0
>>>           Capabilities: [100] Advanced Error Reporting
>>>           Capabilities: [140] Device Serial Number cf-9c-3e-ff-ff-87-d9-c4
> 
>>>      The memory behind bridge should include the 0xfe2000000 address in
>>> card's
>>> BAR0 but it doesn't... I/O and prefetchable memory behind bridge also
> don't
>>> look right. The strange thing is the kernel reassigns the memory behind
>>> bridge's range but never writes it back into the PCI-PCI bridge:
> 
>>> pcieport 0000:00:00.0: BAR 8: assigned [mem 0xfe200000-0xfe2fffff]
>>> pci 0000:01:00.0: BAR 0: assigned [mem 0xfe200000-0xfe201fff 64bit]
> 
>> That does look very odd. This is what I get:
>> root@koelsch:~# lspci -v
>> 00:00.0 PCI bridge: Renesas Technology Corp. Device 001f (prog-if 00
> [Normal decode])
>>          Flags: bus master, fast devsel, latency 0
>>          Bus: primary\0, secondary\x01, subordinate\x01, sec-latency=0
>>          I/O behind bridge: 00001000-00001fff
>>          Memory behind bridge: fe200000-fe2fffff
>>          Prefetchable memory behind bridge: 38000000-380fffff
>>          Capabilities: [40] Power Management version 3
>>          Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>>          Capabilities: [70] Express Root Port (Slot-), MSI 00
>>          Capabilities: [100] Virtual Channel
>>          Capabilities: [1b0] Device Serial Number 00-00-00-00-00-00-00-00
>>
>> 01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
> Connection
>>          Subsystem: Intel Corporation Gigabit CT Desktop Adapter
>>          Flags: bus master, fast devsel, latency 0, IRQ 148
>>          Memory at fe280000 (32-bit, non-prefetchable) [size\x128K]
>>          Memory at fe200000 (32-bit, non-prefetchable) [sizeQ2K]
>>          I/O ports at 1000 [disabled] [size2]
>>          Memory at fe2a0000 (32-bit, non-prefetchable) [size\x16K]
>>          [virtual] Expansion ROM at 38000000 [disabled] [size%6K]
>>          Capabilities: [c8] Power Management version 2
>>          Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>>          Capabilities: [e0] Express Endpoint, MSI 00
>>          Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
>>          Capabilities: [100] Advanced Error Reporting
>>          Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-19-9d-31
>>          Kernel driver in use: e1000e
> 
>     Yes, the bridge looks to be set up correctly here. I've tried to test PCIe
> on the Koelsch board (without the 3.3VAUX resistor soldered though) and
> either
> a PCIe link didn't get detected or the kernel died horrible death.
> Trying to bypass the link check in the driver also resulted in a horrible
> kernel death as described in another thread...
Ok, I'll have a look at bypassing the link check to see what happens.

Cheers
Phil