Serial RapidIO Maintaintance read causes lock up

Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 5023 bytes --]

Hi,

We are currently evaluating Serial RapidIO on two WindRiver SBC8548 boards
that use a Freescale Powerquicc III processor (MPC8548E rev. 2). We are
running U-Boot version 2010.09 as bootloader and are using kernel version
2.6.35.6 stable.

We have consulted multiple resources to collect al the requirements for
a successful RapidIO connection (LAW, TLB, Registers) and we seem to have
configured everything correctly. However, as soon as the board that is
configured as the host starts the enumeration process, the system locks up.
It locks in such a manner that we cannot use a JTAG interface to read any of
the registers.  We have also added a breakpoint just before the command that
causes the lock up, to make sure the registers are correctly set at that
point, and it seems they are.

We have tripple checked everything that we could possibly think of and
everything seems to be configured as required but the system keeps
locking-up so there must be something that we are missing. I really hope
that someone could point us in the right direction. The lock-up occurs when
__fsl_read_rio_config is called by fsl_rio_config_read in fsl-rio.c.

The LAW and TLB entries we have added to U-Boot are as follows:

#define CONFIG_RIO 1
#define CONFIG_SYS_RIO_MEM_VIRT 0xc0000000 /* base address */
#define CONFIG_SYS_RIO_MEM_BUS 0xc0000000 /* base address */
#define CONFIG_SYS_RIO_MEM_PHYS 0xc0000000
#define CONFIG_SYS_RIO_MEM_SIZE 0x20000000 /* 512M */

SET_LAW(CONFIG_SYS_RIO_MEM_PHYS, LAW_SIZE_512M, LAW_TRGT_IF_RIO),

-------------

Here is the kernel log:

Using SBC8548 machine description
Memory CAM mapping: 256 Mb, residual: 0Mb
Linux version 2.6.35.6 (dl704@lxws006) (gcc version 4.1.2 (Wind River Linux
Sourcery G++ 4.1-91)) #7 We
d Sep 29 13:27:18 CEST 2010
bootconsole [udbg0] enabled
setup_arch: bootmem
sbc8548_setup_arch()
arch: exit
Zone PFN ranges:
 DMA      0x00000000 -> 0x00010000
 Normal   empty
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
   0: 0x00000000 -> 0x00010000
MMU: Allocated 1088 bytes of context maps for 255 contexts
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
Kernel command line: root=/dev/nfs rw
nfsroot=192.168.100.21:/thales/target/rfs/sbc8548_wrlinux4
ip=192
.168.100.151:192.168.100.21:192.168.100.21:255.255.255.0:sbc8548_1:eth0:off
console=ttyS0,115200 riohdid=1
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 256884k/262144k available (2712k kernel code, 5260k reserved, 112k
data, 77k bss, 144k init)
Kernel virtual memory layout:
 * 0xfffdf000..0xfffff000  : fixmap
 * 0xfc7f9000..0xfe000000  : early ioremap
 * 0xd1000000..0xfc7f9000  : vmalloc & ioremap
Hierarchical RCU implementation.
       RCU-based detection of stalled CPUs is disabled.
       Verbose stalled-CPUs detection is disabled.
NR_IRQS:512 nr_irqs:512
mpic: Setting up MPIC " OpenPIC  " version 1.2 at e0040000, max 1 CPUs
mpic: ISU size: 80, shift: 7, mask: 7f
mpic: Initializing for 80 sources
clocksource: timebase mult[50cede6] shift[22] registered
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
NET: Registered protocol family 16

PCI: Probing PCI hardware
bio: create slab <bio-0> at 0
vgaarb: loaded
Switching to clocksource timebase
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Of-device full name /soc8548@e0000000
/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
0x0000000020000000.
fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
fsl-of-rio e00c0000.rapidio: DeviceID is 0x1
fsl-of-rio e00c0000.rapidio: Configured as HOST
fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
fsl-of-rio e00c0000.rapidio: Hardware port width: 4
fsl-of-rio e00c0000.rapidio: Training connection status: Four-lane
fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
RIO: enumerate master port 0, RIO0 mport
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: about to trigger '__fsl_read_rio_config'

Regards,
Bastiaan Nijkamp

[-- Attachment #2: Type: text/html, Size: 6059 bytes --]

Re: Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 6487 bytes --]

Hi,

It seems i forgot to include the relevant TLB entries in U-Boot and the
Device Tree in the e-mail, so here they are:

The TLB entries in U-Boot:

/*
 * TLB 3: 256M Non-cacheable, guarded
 * 0xc0000000 256M Rapid IO MEM First half
 */
SET_TLB_ENTRY(1, CONFIG_SYS_RIO_MEM_VIRT, CONFIG_SYS_RIO_MEM_PHYS,
      MAS3_SX|MAS3_SW|MAS3_SR, MAS2_I|MAS2_G,
      0, 3, BOOKE_PAGESZ_256M, 1),

/*
 * TLB 4: 256M Non-cacheable, guarded
 * 0xd0000000 256M Rapid IO MEM Second half
 */
SET_TLB_ENTRY(1, CONFIG_SYS_RIO_MEM_VIRT + 0x10000000,
CONFIG_SYS_RIO_MEM_PHYS + 0x10000000,
      MAS3_SX|MAS3_SW|MAS3_SR, MAS2_I|MAS2_G,
      0, 4, BOOKE_PAGESZ_256M, 1),


And the device tree entry:

 rapidio0:rapidio@c0000 {
           #address-cells = <1>;
           #size-cells = <1>;
           compatible = "fsl,rapidio-delta";
           reg = <0xc0000 0x20000>;
           ranges = <0x0 0xc0000000 0x20000000>;
           interrupt-parent = <&mpic>;
           /* err_irq bell_outb_irq bell_inb_irq
                   msg1_tx_irq msg1_rx_irq msg2_tx_irq msg2_rx_irq */
           interrupts = <0x30 0x2 0x31 0x2 0x32 0x2 0x35 0x2 0x36 0x2 0x37
0x2 0x38 0x2>;
  };

Regards,
Bastiaan Nijkamp

2010/10/2 Bastiaan Nijkamp <bastiaan.nijkamp@gmail.com>

> Hi,
>
> We are currently evaluating Serial RapidIO on two WindRiver SBC8548 boards
> that use a Freescale Powerquicc III processor (MPC8548E rev. 2). We are
> running U-Boot version 2010.09 as bootloader and are using kernel version
> 2.6.35.6 stable.
>
> We have consulted multiple resources to collect al the requirements for
> a successful RapidIO connection (LAW, TLB, Registers) and we seem to have
> configured everything correctly. However, as soon as the board that is
> configured as the host starts the enumeration process, the system locks up.
> It locks in such a manner that we cannot use a JTAG interface to read any of
> the registers.  We have also added a breakpoint just before the command that
> causes the lock up, to make sure the registers are correctly set at that
> point, and it seems they are.
>
> We have tripple checked everything that we could possibly think of and
> everything seems to be configured as required but the system keeps
> locking-up so there must be something that we are missing. I really hope
> that someone could point us in the right direction. The lock-up occurs when
> __fsl_read_rio_config is called by fsl_rio_config_read in fsl-rio.c.
>
> The LAW and TLB entries we have added to U-Boot are as follows:
>
> #define CONFIG_RIO 1
> #define CONFIG_SYS_RIO_MEM_VIRT 0xc0000000 /* base address */
> #define CONFIG_SYS_RIO_MEM_BUS 0xc0000000 /* base address */
> #define CONFIG_SYS_RIO_MEM_PHYS 0xc0000000
> #define CONFIG_SYS_RIO_MEM_SIZE 0x20000000 /* 512M */
>
> SET_LAW(CONFIG_SYS_RIO_MEM_PHYS, LAW_SIZE_512M, LAW_TRGT_IF_RIO),
>
> -------------
>
> Here is the kernel log:
>
> Using SBC8548 machine description
> Memory CAM mapping: 256 Mb, residual: 0Mb
> Linux version 2.6.35.6 (dl704@lxws006) (gcc version 4.1.2 (Wind River
> Linux Sourcery G++ 4.1-91)) #7 We
> d Sep 29 13:27:18 CEST 2010
> bootconsole [udbg0] enabled
> setup_arch: bootmem
> sbc8548_setup_arch()
> arch: exit
> Zone PFN ranges:
>  DMA      0x00000000 -> 0x00010000
>  Normal   empty
> Movable zone start PFN for each node
> early_node_map[1] active PFN ranges
>    0: 0x00000000 -> 0x00010000
> MMU: Allocated 1088 bytes of context maps for 255 contexts
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
> Kernel command line: root=/dev/nfs rw nfsroot=192.168.100.21:/thales/target/rfs/sbc8548_wrlinux4
> ip=192
> .168.100.151:192.168.100.21:192.168.100.21:255.255.255.0:sbc8548_1:eth0:off
> console=ttyS0,115200 riohdid=1
> PID hash table entries: 1024 (order: 0, 4096 bytes)
> Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
> Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
> Memory: 256884k/262144k available (2712k kernel code, 5260k reserved, 112k
> data, 77k bss, 144k init)
> Kernel virtual memory layout:
>  * 0xfffdf000..0xfffff000  : fixmap
>  * 0xfc7f9000..0xfe000000  : early ioremap
>  * 0xd1000000..0xfc7f9000  : vmalloc & ioremap
> Hierarchical RCU implementation.
>        RCU-based detection of stalled CPUs is disabled.
>        Verbose stalled-CPUs detection is disabled.
> NR_IRQS:512 nr_irqs:512
> mpic: Setting up MPIC " OpenPIC  " version 1.2 at e0040000, max 1 CPUs
> mpic: ISU size: 80, shift: 7, mask: 7f
> mpic: Initializing for 80 sources
> clocksource: timebase mult[50cede6] shift[22] registered
> pid_max: default: 32768 minimum: 301
> Mount-cache hash table entries: 512
> NET: Registered protocol family 16
>
> PCI: Probing PCI hardware
> bio: create slab <bio-0> at 0
> vgaarb: loaded
> Switching to clocksource timebase
> NET: Registered protocol family 2
> IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
> TCP established hash table entries: 8192 (order: 4, 65536 bytes)
> TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
> TCP: Hash tables configured (established 8192 bind 8192)
> TCP reno registered
> UDP hash table entries: 256 (order: 0, 4096 bytes)
> UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
> NET: Registered protocol family 1
> RPC: Registered udp transport module.
> RPC: Registered tcp transport module.
> RPC: Registered tcp NFSv4.1 backchannel transport module.
> Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
> fsl-of-rio e00c0000.rapidio: Of-device full name /soc8548@e0000000
> /rapidio@c0000
> fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
> fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
> 0x0000000020000000.
> fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
> fsl-of-rio e00c0000.rapidio: DeviceID is 0x1
> fsl-of-rio e00c0000.rapidio: Configured as HOST
> fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
> fsl-of-rio e00c0000.rapidio: Hardware port width: 4
> fsl-of-rio e00c0000.rapidio: Training connection status: Four-lane
> fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
> RIO: enumerate master port 0, RIO0 mport
> fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
> fsl_rio_config_read: Passed IS_ALIGNED.
> fsl_rio_config_read: Passed 'out_be32_1'
> fsl_rio_config_read: Passed 'out_be32_2'
> fsl_rio_config_read: len is 4
> fsl_rio_config_read: about to trigger '__fsl_read_rio_config'
>
> Regards,
>  Bastiaan Nijkamp
>

[-- Attachment #2: Type: text/html, Size: 8642 bytes --]

Re: Serial RapidIO Maintaintance read causes lock up

[Micha Nelissen]

Hi Bastian,

Bastiaan Nijkamp wrote:
> It seems i forgot to include the relevant TLB entries in U-Boot and the 
> Device Tree in the e-mail, so here they are:
> 
> The TLB entries in U-Boot:

The kernel maintains the TLB, you must not set those in U-boot. It might 
cause conflicts when the kernel chooses its virtual memory space. You 
should only configure LAWs in U-boot as the kernel does not do that. 
That's the physical address you pass in the DTB (which seems to work, 
reading your kernel log).

Do you access RapidIO space in U-boot also?

Do you have a logic analyser, then you can see whether the read is 
actually coming out.

Check whether the time-to-live, packet and link timeouts have been set 
to sane values to prevent deadlocks (especially time-to-live). At least 
then your kernel will crash instead of lock up.

Micha

RE: Serial RapidIO Maintaintance read causes lock up

[Bounine, Alexandre]

Bastiaan Nijkamp <bastiaan.nijkamp@gmail.com> wrote:

>>=A0How the host ID is set on your host board?
>> Normally rio_enum_host() should increment next_destid in your case.
>
> The hostID is set to 0x0 with the riohdid parameter as boot =
argument.=A0
>

In this case I would expect to see ID=3D1 assigned to the endpoint. I =
will take a look based on your use scenario. Let me know if you find the =
source of problem before I get back to you about this issue.=20

>>=A0Make sure that you have the MASTER bit is set in agent's GCCSR =
register (0xC_013C).
>> If your board uses HW config switches to set host/agent mode this bit =
will be 0 for
>> agent.
>> For quick test you may keep both boards in the host mode - the =
current RIO implementation >> relies on "riohdid=3D" command line =
parameter instead of HOST bit.
>
> This seems to have done the trick :-) However, i wonder, doesn't it =
make more sense to
> make the driver check this setting and correcting it or giving an =
error instead of
> entering a endless loop?=A0

One of my recent patches handles the MASTER bit. It is in -mm tree now.

In the case that you have seen, the agent does not enter endless loop - =
it just has request that cannot leave the SRIO controller. As result =
that request cannot be timed out and hangs the processor (no machine =
check).

Alex.
 =20

Re: Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 1670 bytes --]

> How the host ID is set on your host board?
Normally rio_enum_host() should increment next_destid in your case.

The hostID is set to 0x0 with the riohdid parameter as boot argument.

> Make sure that you have the MASTER bit is set in agent's GCCSR register
(0xC_013C).
If your board uses HW config switches to set host/agent mode this bit will
be 0 for agent.
For quick test you may keep both boards in the host mode - the current RIO
implementation relies on "riohdid=" command line parameter instead of HOST
bit.

This seems to have done the trick :-) However, i wonder, doesn't it make
more sense to make the driver check this setting and correcting it or giving
an error instead of entering a endless loop?

Thank you for helping out, it is very much appreciated.

Bastiaan.


2010/10/13 Bounine, Alexandre <Alexandre.Bounine@idt.com>

> Bastiaan Nijkamp wrote:
>
> >Has the driver ever been tested/used without a switch attached? Because
> when the host >(which has ID 0x0) enumerates the other board it also assigns
> ID 0x0 to the agent, it seems >that the agent should have been assigned 0x1
> as ID.
>
> How the host ID is set on your host board?
> Normally rio_enum_host() should increment next_destid in your case.
>
> >Another thing is that the agent is now hanging on the discovery process.
>
> Make sure that you have the MASTER bit is set in agent's GCCSR register
> (0xC_013C).
> If your board uses HW config switches to set host/agent mode this bit will
> be 0 for agent.
> For quick test you may keep both boards in the host mode - the current RIO
> implementation relies on "riohdid=" command line parameter instead of HOST
> bit.
>
>
> Alex.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2279 bytes --]

RE: Serial RapidIO Maintaintance read causes lock up

[Bounine, Alexandre]

Bastiaan Nijkamp wrote:
=A0
>Has the driver ever been tested/used without a switch attached? Because =
when the host >(which has ID 0x0) enumerates the other board it also =
assigns ID 0x0 to the agent, it seems >that the agent should have=A0been =
assigned 0x1 as ID.

How the host ID is set on your host board?
Normally rio_enum_host() should increment next_destid in your case.=20

>Another thing is that the agent is now hanging on the discovery =
process.

Make sure that you have the MASTER bit is set in agent's GCCSR register =
(0xC_013C).
If your board uses HW config switches to set host/agent mode this bit =
will be 0 for agent.
For quick test you may keep both boards in the host mode - the current =
RIO implementation relies on "riohdid=3D" command line parameter instead =
of HOST bit.


Alex.
=20
=A0

Re: Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 2504 bytes --]

Hi,

I want to rectify my last e-mail. It seemed to be a weird bug in the tool
suite that we are using, since it would be impossible that all the read-only
registers also had that same strange value, it did not happen again
either. I do have another question, however.

Has the driver ever been tested/used without a switch attached? Because when
the host (which has ID 0x0) enumerates the other board it also assigns ID
0x0 to the agent, it seems that the agent should have been assigned 0x1 as
ID. Another thing is that the agent is now hanging on the discovery process.

Regards,
Bastiaan Nijkamp

2010/10/11 Bastiaan Nijkamp <bastiaan.nijkamp@gmail.com>

> Hi,
>
> We have found a poorly documented jumper on our boards that force pci-mode
> to 32bit instead of 64bit. It seems to have an effect on RapidIO aswell
> since the host now completes the enumeration process including finding the
> other board. However, as soon as the agent starts the peer discovery
> process, ALL RapidIO related registers on the agent are set to 0xFF4A.
> Which, offcourse, caused unpredicted behaviour. All other jumpers are set as
> they should be, especially the reference clock (100Mhz) and linkspeed
> settings (1.25Gbps). I am currently having some trouble understanding why
> this happens.
>
> I've double checked Accept All on both boards in the registers and on both
> boards it has been set correctly before discovery/enumeration.
>
> Here is the LAW configuration from u-boot for the agent and host:
>
> Local Access Window Configuration
> LAWBAR00: 0x00000000 LAWAR0x00: 0x80f0001b
>         (EN: 1 TGT: 0x0f SIZE: 256 MiB)
> LAWBAR01: 0x00080000 LAWAR0x01: 0x8000001c
>         (EN: 1 TGT: 0x00 SIZE: 512 MiB)
> LAWBAR02: 0x000e2000 LAWAR0x02: 0x80000016
>         (EN: 1 TGT: 0x00 SIZE: 8 MiB)
> LAWBAR03: 0x000f0000 LAWAR0x03: 0x8040001b
>         (EN: 1 TGT: 0x04 SIZE: 256 MiB)
> LAWBAR04: 0x000c0000 LAWAR0x04: 0x80c0001c
>         (EN: 1 TGT: 0x0c SIZE: 512 MiB)
> LAWBAR05: 0x00000000 LAWAR0x05: 0x00000000
>         (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
> LAWBAR06: 0x00000000 LAWAR0x06: 0x00000000
>         (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
> LAWBAR07: 0x00000000 LAWAR0x07: 0x00000000
>         (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
> LAWBAR08: 0x00000000 LAWAR0x08: 0x00000000
>         (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
> LAWBAR09: 0x00000000 LAWAR0x09: 0x00000000
>         (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
>
> We have removed the RapidIO TLB Entries from u-boot.
>
> Kind regards,
> Bastiaan Nijkamp
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3304 bytes --]

Re: Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

Hi,

We have found a poorly documented jumper on our boards that force pci-mode
to 32bit instead of 64bit. It seems to have an effect on RapidIO aswell
since the host now completes the enumeration process including finding the
other board. However, as soon as the agent starts the peer discovery
process, ALL RapidIO related registers on the agent are set to 0xFF4A.
Which, offcourse, caused unpredicted behaviour. All other jumpers are set as
they should be, especially the reference clock (100Mhz) and linkspeed
settings (1.25Gbps). I am currently having some trouble understanding why
this happens.

I've double checked Accept All on both boards in the registers and on both
boards it has been set correctly before discovery/enumeration.

Here is the LAW configuration from u-boot for the agent and host:

Local Access Window Configuration
LAWBAR00: 0x00000000 LAWAR0x00: 0x80f0001b
        (EN: 1 TGT: 0x0f SIZE: 256 MiB)
LAWBAR01: 0x00080000 LAWAR0x01: 0x8000001c
        (EN: 1 TGT: 0x00 SIZE: 512 MiB)
LAWBAR02: 0x000e2000 LAWAR0x02: 0x80000016
        (EN: 1 TGT: 0x00 SIZE: 8 MiB)
LAWBAR03: 0x000f0000 LAWAR0x03: 0x8040001b
        (EN: 1 TGT: 0x04 SIZE: 256 MiB)
LAWBAR04: 0x000c0000 LAWAR0x04: 0x80c0001c
        (EN: 1 TGT: 0x0c SIZE: 512 MiB)
LAWBAR05: 0x00000000 LAWAR0x05: 0x00000000
        (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBAR06: 0x00000000 LAWAR0x06: 0x00000000
        (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBAR07: 0x00000000 LAWAR0x07: 0x00000000
        (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBAR08: 0x00000000 LAWAR0x08: 0x00000000
        (EN: 0 TGT: 0x00 SIZE: 2 Bytes)
LAWBAR09: 0x00000000 LAWAR0x09: 0x00000000
        (EN: 0 TGT: 0x00 SIZE: 2 Bytes)

We have removed the RapidIO TLB Entries from u-boot.

Kind regards,
Bastiaan Nijkamp

[-- Attachment #2: Type: text/html, Size: 2202 bytes --]

RE: Serial RapidIO Maintaintance read causes lock up

[Anderson, Trevor]

[-- Attachment #1: Type: text/plain, Size: 14335 bytes --]

I may have come across a similar problem, but I've never worked card-to-card without at least one switch in the way.
The problems I have encountered have been hang-ups during memory-mapped maintenance reads to devices that the switch reports as "up".

A workaround for the hang-up is to use a DMA transfer that bypasses the ATMU to perform a maintenance read (not all, just the first, a test read of a new discovery). If the DMA fails, as it does in those unusual circumstances, it will report a bus transfer error - but it will not hang up. You can recover, and avoid or re-try the connection later.

A real "fix" for your problem may be to ensure that both of your RapidIO interfaces are programmed to accept all incoming transactions (see "Accept All Configuration Register (AACR)" in Freescale reference manual). But no guarantees with that - it just seemed to clear up problems on my own rig.





From: linuxppc-dev-bounces+tanderson=curtisswright.com@lists.ozlabs.org [mailto:linuxppc-dev-bounces+tanderson=curtisswright.com@lists.ozlabs.org] On Behalf Of Bastiaan Nijkamp
Sent: Tuesday, October 05, 2010 7:28 AM
To: John Traill
Cc: Bounine, Alexandre; linuxppc-dev@lists.ozlabs.org
Subject: Re: Serial RapidIO Maintaintance read causes lock up

Hi John,

1. Yes, they are both running the exact same kernel and both are configured in the same way. With the exception that one is set as host and the other as a agent.

2. Accept All is set for both boards.

3. As i understand, the agent cannot send anything before it is enumerated, so it would be safe to first reset the agent and right after that the host. In either case, thats the way i am using. The full kernel log until the discovery times out after 30 seconds is shown below:

Using SBC8548 machine description
Memory CAM mapping: 256 Mb, residual: 0Mb
Linux version 2.6.35.6 (dl704@lxws006<mailto:dl704@lxws006>) (gcc version 4.4.1 (Wind River Linux Sour
cery G++ 4.4-250) ) #3 Tue Oct 5 13:24:45 CEST 2010
bootconsole [udbg0] enabled
setup_arch: bootmem
sbc8548_setup_arch()
arch: exit
Zone PFN ranges:
  DMA      0x00000000 -> 0x00010000
  Normal   empty
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00010000
MMU: Allocated 1088 bytes of context maps for 255 contexts
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
Kernel command line: root=/dev/nfs rw nfsroot=192.168.100.21:/thales/target/rfs/
sbc8548_wrlinux4 ip=192.168.100.151:192.168.100.21:192.168.100.21:255.255.255.0:
sbc8548_1:eth0:off console=ttyS0,115200
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 256996k/262144k available (2644k kernel code, 5148k reserved, 108k data,
 77k bss, 136k init)
Kernel virtual memory layout:
  * 0xfffdf000..0xfffff000  : fixmap
  * 0xfdffd000..0xfe000000  : early ioremap
  * 0xd1000000..0xfdffd000  : vmalloc & ioremap
Hierarchical RCU implementation.
 RCU-based detection of stalled CPUs is disabled.
 Verbose stalled-CPUs detection is disabled.
NR_IRQS:512 nr_irqs:512
mpic: Setting up MPIC " OpenPIC  " version 1.2 at e0040000, max 1 CPUs
mpic: ISU size: 80, shift: 7, mask: 7f
mpic: Initializing for 80 sources
clocksource: timebase mult[50cede6] shift[22] registered
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
NET: Registered protocol family 16

PCI: Probing PCI hardware
bio: create slab <bio-0> at 0
vgaarb: loaded
Switching to clocksource timebase
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Of-device full name /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size 0x0000000020000000.
fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
fsl-of-rio e00c0000.rapidio: DeviceID is 0xffffffff
fsl-of-rio e00c0000.rapidio: Configured as AGENT
fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
fsl-of-rio e00c0000.rapidio: Hardware port width: 4
fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO Maintainance Window Size 0x400000,New Main Start: 0xd1080000
RIO: discover master port 0, RIO0 mport

A interesting thing that i found out is that when the agent is reset while the host is locked up (eg. it cannot be stopped nor can i read the registers and memory trough a JTAG Interface), the host comes back online and just continues booting linux with a RapidIO error. See the log below.

Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Of-device full name /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size 0x0000000020000000.
fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
fsl-of-rio e00c0000.rapidio: DeviceID is 0x0
fsl-of-rio e00c0000.rapidio: Configured as HOST
fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
fsl-of-rio e00c0000.rapidio: Hardware port width: 4
fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO Maintainance Window Size 0x400000,New Main Start: 0xd1080000
RIO: enumerate master port 0, RIO0 mport
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d1080068
RIO: cfg_read error -14 for ff:0:68
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d1080068
RIO: cfg_read error -14 for ff:0:68
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d1080068
RIO: cfg_read error -14 for ff:0:68
RIO: master port 0 device has lost enumeration to a remote host

Regards,
Bastiaan
2010/10/5 John Traill <john.traill@freescale.com<mailto:john.traill@freescale.com>>
Bastiaan,

A few things to check.

1. Is the target board also set up for small common transport system size ie 256.

2. Make sure the target has "Accept All" set - in fsl_rio.c look for
      /* Set to receive any dist ID for serial RapidIO controller. */
       if (port->phy_type == RIO_PHY_SERIAL)
               out_be32((priv->regs_win + RIO_ISR_AACR), RIO_ISR_AACR_AA);

3. How do you synchronise reset between both systems ? Both need to be reset to insure the inbound/outbound ackid's remain in sync. If you only reset one then you have the potential for the ackid's to get out of sync. Also what is the kernel log on the agent system ?

Cheers.



On 05/10/10 09:56, Bastiaan Nijkamp wrote:

Hi Alex,

Thanks for your advice. We are trying to make a board-to-board
connection without any additional hardware (eg. a switch). The boards
use a 50-pin, right-angle MEC8-125-02-L-D-RA1 connector from SAMTEC and
are connected trough a EEDP-016-12.00-RA1-RA2-2 cross cable from SAMTEC.
I hope this information is sufficient since there is not much one can
find about it on Google. In addition, you can see a picture of the board
including the connector in the datasheet located at
http://www.windriver.com/products/product-notes/SBC8548E-product-note.pdf.
It is the connector on the left side of the PCI-EX slot.

We have tried your suggestion but the situation does not change other
than the lane-mode being set to single lane 0, it still locks up when
trying to generate a maintenance transaction. I still think it is memory
related since the lock up occurs when accessing the maintenance window.
Although all memory related settings seems to be alright.

The kernel output is as follows:

Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Of-device full name
/soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
0x0000000010000000.
fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
fsl-of-rio e00c0000.rapidio: DeviceID is 0x0
fsl-of-rio e00c0000.rapidio: Configured as HOST
fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
fsl-of-rio e00c0000.rapidio: Hardware port width: 4
fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO
Maintainance Window Size 0x400000,New Main Start: 0xd1080000
RIO: enumerate master port 0, RIO0 mport
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d108006

Regards,
Bastiaan

2010/10/4 Bounine, Alexandre <Alexandre.Bounine@idt.com<mailto:Alexandre.Bounine@idt.com>
<mailto:Alexandre.Bounine@idt.com<mailto:Alexandre.Bounine@idt.com>>>


   Hi Bastiaan,

   Are you trying board-to-board connection?
   I am not familiar with WRS SBC8548 board - which type of connector they
   use for SRIO?

   Assuming that all configuration is correct,
   I would recommend first to try setting up x1 link mode at the lowest
   link speed.
   The x4 mode may present challenges in some cases.

   For quick test you may just add port width override into fsl_rio.c
   like shown below (ugly but sometimes it helps ;) ):

   @@ -1461,10 +1461,16 @@ int fsl_rio_setup(struct platform_device *dev)
           rio_register_mport(port);

           priv->regs_win = ioremap(regs.start, regs.end - regs.start + 1);
           rio_regs_win = priv->regs_win;

   +dev_info(&dev->dev, "Overriding RIO_PORT setting to single lane 0\n");
   +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
   0x800000);
   +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
   0x2000000);
   +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) &
   ~0x800000);
   +msleep(100);
   +
           /* Probe the master port phy type */
           ccsr = in_be32(priv->regs_win + RIO_CCSR);
           port->phy_type = (ccsr & 1) ? RIO_PHY_SERIAL : RIO_PHY_PARALLEL;
           dev_info(&dev->dev, "RapidIO PHY type: %s\n",
                           (port->phy_type == RIO_PHY_PARALLEL) ?
   "parallel" :


   Let me know what happens.
   Please keep me in the CC: list next time when posting RapidIO questions
   to the linuxppc-dev or kernel mailing lists.

   Regards,

   Alex.



_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org<mailto:Linuxppc-dev@lists.ozlabs.org>
https://lists.ozlabs.org/listinfo/linuxppc-dev

--
John Traill
Systems Engineer
Network and Computing Systems Group

Freescale Semiconductor UK LTD
Colvilles Road
East Kilbride
Glasgow G75 0TG, Scotland

Tel: +44 (0) 1355 355494
Fax: +44 (0) 1355 261790

E-mail: john.traill@freescale.com<mailto:john.traill@freescale.com>

Registration Number: SC262720
VAT Number: GB831329053

[ ] General Business Use
[ ] Freescale Internal Use Only
[ ] Freescale Confidential Proprietary


_______________________________________________________________________
This e-mail and any files transmitted with it are proprietary and intended solely for the use of the individual or entity to whom they are addressed. If you have reason to believe that you have received this e-mail in error, please notify the sender and destroy this email and any attached files. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of the Curtiss-Wright Corporation or any of its subsidiaries.  Documents attached hereto may contain technology subject to government export regulations. Recipient is solely responsible for ensuring that any re-export, transfer or disclosure of this information is in accordance with applicable government export regulations.  The recipient should check this e-mail and any attachments for the presence of viruses. Curtiss-Wright Corporation and its subsidiaries accept no liability for any damage caused by any virus transmitted by this e-mail.

[-- Attachment #2: Type: text/html, Size: 24406 bytes --]

RE: Serial RapidIO Maintaintance read causes lock up

[Bounine, Alexandre]

Bastiaan Nijkamp <bastiaan.nijkamp@gmail.com> wrote:=20
=A0
>A interesting thing that i found out is that when the agent is reset =
while the host is >locked up (eg. it cannot be stopped nor can i read =
the registers and memory trough a JTAG >Interface), the host comes back =
online and just continues booting linux with a RapidIO >error. See the =
log below.
... skip ...=A0
>fsl_rio_config_read: going to request to read data at d1080068
>RIO: cfg_read error -14 for ff:0:68
... skip ...
>RIO: master port 0 device has lost enumeration to a remote host

Looks like during agent reset your SRIO link becomes good and host's =
requests go ahead.
After that you get a machine check (response time out): link is OK but =
agent is not configured.

Can you check/print 0xC_0148 and 0xC_0158 registers of SRIO block before =
the first maintenance read.

Alex.
=20

Re: Serial RapidIO Maintaintance read causes lock up

[John Traill]

Bastiaan,

On 05/10/10 15:28, Bastiaan Nijkamp wrote:
> Hi John,
> 1. Yes, they are both running the exact same kernel and both are
> configured in the same way. With the exception that one is set as host
> and the other as a agent.
> 2. Accept All is set for both boards.
> 3. As i understand, the agent cannot send anything before it is
> enumerated, so it would be safe to first reset the agent and right after
> that the host. In either case, thats the way i am using. The full kernel
> log until the discovery times out after 30 seconds is shown below:
> Using SBC8548 machine description
<snip>

In which case could you check the lawbar initialisation in u-boot for conflicts. 
Would you post a dump of the lawbar registers from uboot.

The TLB's aren't important in this case as linux will reprogram as required.

Cheers

-- 
John Traill
Systems Engineer
Network and Computing Systems Group

Freescale Semiconductor UK LTD
Colvilles Road
East Kilbride
Glasgow G75 0TG, Scotland

Tel: +44 (0) 1355 355494
Fax: +44 (0) 1355 261790

E-mail: john.traill@freescale.com

Registration Number: SC262720
VAT Number: GB831329053

[ ] General Business Use
[ ] Freescale Internal Use Only
[ ] Freescale Confidential Proprietary

Re: Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

Hi Alex,

Yes, Sorry, that was a typo. The correct address that is printed is
0xd1080068.
And i did not disable the error handler, CONFIG_E500 is set at compile time.

Regards,
Bastiaan
2010/10/5 Bounine, Alexandre <Alexandre.Bounine@idt.com>

> Hi Bastiaan,
>
>
> Bastiaan Nijkamp <bastiaan.nijkamp@gmail.com> wrote:
>
> >fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO
> Maintainance Window Size >0x400000,New Main Start: 0xd1080000
> >RIO: enumerate master port 0, RIO0 mport
> >fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len
> 4
> ... skip ....
> >fsl_rio_config_read: triggering '__fsl_read_rio_config'
> >fsl_rio_config_read: going to request to read data at d108006
>
> An address printed in the last line looks strange - is this typo or real
> value?
> I would expect to see d1080068 here if your maintenance window starts at
> 0xd1080000.
>
> Alex.
>

[-- Attachment #2: Type: text/html, Size: 1392 bytes --]

Re: Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 11945 bytes --]

Hi John,

1. Yes, they are both running the exact same kernel and both are configured
in the same way. With the exception that one is set as host and the other as
a agent.

2. Accept All is set for both boards.

3. As i understand, the agent cannot send anything before it is enumerated,
so it would be safe to first reset the agent and right after that the host.
In either case, thats the way i am using. The full kernel log until the
discovery times out after 30 seconds is shown below:

Using SBC8548 machine description
Memory CAM mapping: 256 Mb, residual: 0Mb
Linux version 2.6.35.6 (dl704@lxws006) (gcc version 4.4.1 (Wind River Linux
Sour
cery G++ 4.4-250) ) #3 Tue Oct 5 13:24:45 CEST 2010
bootconsole [udbg0] enabled
setup_arch: bootmem
sbc8548_setup_arch()
arch: exit
Zone PFN ranges:
  DMA      0x00000000 -> 0x00010000
  Normal   empty
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00010000
MMU: Allocated 1088 bytes of context maps for 255 contexts
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
Kernel command line: root=/dev/nfs rw nfsroot=192.168.100.21:
/thales/target/rfs/
sbc8548_wrlinux4 ip=192.168.100.151:192.168.100.21:192.168.100.21:255
.255.255.0:
sbc8548_1:eth0:off console=ttyS0,115200
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 256996k/262144k available (2644k kernel code, 5148k reserved, 108k
data,
 77k bss, 136k init)
Kernel virtual memory layout:
  * 0xfffdf000..0xfffff000  : fixmap
  * 0xfdffd000..0xfe000000  : early ioremap
  * 0xd1000000..0xfdffd000  : vmalloc & ioremap
Hierarchical RCU implementation.
 RCU-based detection of stalled CPUs is disabled.
 Verbose stalled-CPUs detection is disabled.
NR_IRQS:512 nr_irqs:512
mpic: Setting up MPIC " OpenPIC  " version 1.2 at e0040000, max 1 CPUs
mpic: ISU size: 80, shift: 7, mask: 7f
mpic: Initializing for 80 sources
clocksource: timebase mult[50cede6] shift[22] registered
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
NET: Registered protocol family 16

PCI: Probing PCI hardware
bio: create slab <bio-0> at 0
vgaarb: loaded
Switching to clocksource timebase
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
UDP hash table entries: 256 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
NET: Registered protocol family 1
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Of-device full name
/soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
0x0000000020000000.
fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
fsl-of-rio e00c0000.rapidio: DeviceID is 0xffffffff
fsl-of-rio e00c0000.rapidio: Configured as AGENT
fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
fsl-of-rio e00c0000.rapidio: Hardware port width: 4
fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO Maintainance
Window Size 0x400000,New Main Start: 0xd1080000
RIO: discover master port 0, RIO0 mport

A interesting thing that i found out is that when the agent is reset while
the host is locked up (eg. it cannot be stopped nor can i read the registers
and memory trough a JTAG Interface), the host comes back online and just
continues booting linux with a RapidIO error. See the log below.

Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Of-device full name
/soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
0x0000000020000000.
fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
fsl-of-rio e00c0000.rapidio: DeviceID is 0x0
fsl-of-rio e00c0000.rapidio: Configured as HOST
fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
fsl-of-rio e00c0000.rapidio: Hardware port width: 4
fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO Maintainance
Window Size 0x400000,New Main Start: 0xd1080000
RIO: enumerate master port 0, RIO0 mport
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d1080068
RIO: cfg_read error -14 for ff:0:68
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d1080068
RIO: cfg_read error -14 for ff:0:68
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d1080068
RIO: cfg_read error -14 for ff:0:68
RIO: master port 0 device has lost enumeration to a remote host

Regards,
Bastiaan
2010/10/5 John Traill <john.traill@freescale.com>

> Bastiaan,
>
> A few things to check.
>
> 1. Is the target board also set up for small common transport system size
> ie 256.
>
> 2. Make sure the target has "Accept All" set - in fsl_rio.c look for
>
>>       /* Set to receive any dist ID for serial RapidIO controller. */
>>        if (port->phy_type == RIO_PHY_SERIAL)
>>                out_be32((priv->regs_win + RIO_ISR_AACR), RIO_ISR_AACR_AA);
>>
>
> 3. How do you synchronise reset between both systems ? Both need to be
> reset to insure the inbound/outbound ackid's remain in sync. If you only
> reset one then you have the potential for the ackid's to get out of sync.
> Also what is the kernel log on the agent system ?
>
> Cheers.
>
>
>
> On 05/10/10 09:56, Bastiaan Nijkamp wrote:
>
>>
>> Hi Alex,
>>
>> Thanks for your advice. We are trying to make a board-to-board
>> connection without any additional hardware (eg. a switch). The boards
>> use a 50-pin, right-angle MEC8-125-02-L-D-RA1 connector from SAMTEC and
>> are connected trough a EEDP-016-12.00-RA1-RA2-2 cross cable from SAMTEC.
>> I hope this information is sufficient since there is not much one can
>> find about it on Google. In addition, you can see a picture of the board
>> including the connector in the datasheet located at
>> http://www.windriver.com/products/product-notes/SBC8548E-product-note.pdf
>> .
>> It is the connector on the left side of the PCI-EX slot.
>>
>> We have tried your suggestion but the situation does not change other
>> than the lane-mode being set to single lane 0, it still locks up when
>> trying to generate a maintenance transaction. I still think it is memory
>> related since the lock up occurs when accessing the maintenance window.
>> Although all memory related settings seems to be alright.
>>
>> The kernel output is as follows:
>>
>> Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
>> fsl-of-rio e00c0000.rapidio: Of-device full name
>> /soc8548@e0000000/rapidio@c0000
>> fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
>> fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
>> 0x0000000010000000.
>> fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
>> fsl-of-rio e00c0000.rapidio: DeviceID is 0x0
>> fsl-of-rio e00c0000.rapidio: Configured as HOST
>> fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
>> fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
>> fsl-of-rio e00c0000.rapidio: Hardware port width: 4
>> fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
>> fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
>> fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO
>> Maintainance Window Size 0x400000,New Main Start: 0xd1080000
>> RIO: enumerate master port 0, RIO0 mport
>> fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
>> fsl_rio_config_read: Passed IS_ALIGNED.
>> fsl_rio_config_read: Passed 'out_be32_1'
>> fsl_rio_config_read: Passed 'out_be32_2'
>> fsl_rio_config_read: len is 4
>> fsl_rio_config_read: triggering '__fsl_read_rio_config'
>> fsl_rio_config_read: going to request to read data at d108006
>>
>> Regards,
>> Bastiaan
>>
>> 2010/10/4 Bounine, Alexandre <Alexandre.Bounine@idt.com
>> <mailto:Alexandre.Bounine@idt.com>>
>>
>>
>>    Hi Bastiaan,
>>
>>    Are you trying board-to-board connection?
>>    I am not familiar with WRS SBC8548 board - which type of connector they
>>    use for SRIO?
>>
>>    Assuming that all configuration is correct,
>>    I would recommend first to try setting up x1 link mode at the lowest
>>    link speed.
>>    The x4 mode may present challenges in some cases.
>>
>>    For quick test you may just add port width override into fsl_rio.c
>>    like shown below (ugly but sometimes it helps ;) ):
>>
>>    @@ -1461,10 +1461,16 @@ int fsl_rio_setup(struct platform_device *dev)
>>            rio_register_mport(port);
>>
>>            priv->regs_win = ioremap(regs.start, regs.end - regs.start +
>> 1);
>>            rio_regs_win = priv->regs_win;
>>
>>    +dev_info(&dev->dev, "Overriding RIO_PORT setting to single lane 0\n");
>>    +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
>>    0x800000);
>>    +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
>>    0x2000000);
>>    +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) &
>>    ~0x800000);
>>    +msleep(100);
>>    +
>>            /* Probe the master port phy type */
>>            ccsr = in_be32(priv->regs_win + RIO_CCSR);
>>            port->phy_type = (ccsr & 1) ? RIO_PHY_SERIAL :
>> RIO_PHY_PARALLEL;
>>            dev_info(&dev->dev, "RapidIO PHY type: %s\n",
>>                            (port->phy_type == RIO_PHY_PARALLEL) ?
>>    "parallel" :
>>
>>
>>    Let me know what happens.
>>    Please keep me in the CC: list next time when posting RapidIO questions
>>    to the linuxppc-dev or kernel mailing lists.
>>
>>    Regards,
>>
>>    Alex.
>>
>>
>>
>>
>> _______________________________________________
>> Linuxppc-dev mailing list
>> Linuxppc-dev@lists.ozlabs.org
>> https://lists.ozlabs.org/listinfo/linuxppc-dev
>>
>
> --
> John Traill
> Systems Engineer
> Network and Computing Systems Group
>
> Freescale Semiconductor UK LTD
> Colvilles Road
> East Kilbride
> Glasgow G75 0TG, Scotland
>
> Tel: +44 (0) 1355 355494
> Fax: +44 (0) 1355 261790
>
> E-mail: john.traill@freescale.com
>
> Registration Number: SC262720
> VAT Number: GB831329053
>
> [ ] General Business Use
> [ ] Freescale Internal Use Only
> [ ] Freescale Confidential Proprietary
>
>

[-- Attachment #2: Type: text/html, Size: 14103 bytes --]

RE: Serial RapidIO Maintaintance read causes lock up

[Bounine, Alexandre]

Hi John,

John Traill <john.traill@freescale.com> wrote:
> 2. Make sure the target has "Accept All" set - in fsl_rio.c look for
> >        /* Set to receive any dist ID for serial RapidIO controller.
*/
> >         if (port->phy_type =3D=3D RIO_PHY_SERIAL)
> >                 out_be32((priv->regs_win + RIO_ISR_AACR),
RIO_ISR_AACR_AA);
>

Looks like in Bastiaan's case request cannot leave the SRIO controller.
Otherwise he would get a machine check if a response is timed out
(assuming that he did not disable it).

Alex.
=20
=20

RE: Serial RapidIO Maintaintance read causes lock up

[Bounine, Alexandre]

Hi Bastiaan,
=20

Bastiaan Nijkamp <bastiaan.nijkamp@gmail.com> wrote:=20

>fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO
Maintainance Window Size >0x400000,New Main Start: 0xd1080000
>RIO: enumerate master port 0, RIO0 mport
>fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len
4
... skip ....
>fsl_rio_config_read: triggering '__fsl_read_rio_config'
>fsl_rio_config_read: going to request to read data at d108006

An address printed in the last line looks strange - is this typo or real
value?
I would expect to see d1080068 here if your maintenance window starts at
0xd1080000.

Alex.

Re: Serial RapidIO Maintaintance read causes lock up

[John Traill]

Bastiaan,

A few things to check.

1. Is the target board also set up for small common transport system size ie 256.

2. Make sure the target has "Accept All" set - in fsl_rio.c look for
>        /* Set to receive any dist ID for serial RapidIO controller. */
>         if (port->phy_type == RIO_PHY_SERIAL)
>                 out_be32((priv->regs_win + RIO_ISR_AACR), RIO_ISR_AACR_AA);

3. How do you synchronise reset between both systems ? Both need to be reset to 
insure the inbound/outbound ackid's remain in sync. If you only reset one then 
you have the potential for the ackid's to get out of sync. Also what is the 
kernel log on the agent system ?

Cheers.


On 05/10/10 09:56, Bastiaan Nijkamp wrote:
>
> Hi Alex,
>
> Thanks for your advice. We are trying to make a board-to-board
> connection without any additional hardware (eg. a switch). The boards
> use a 50-pin, right-angle MEC8-125-02-L-D-RA1 connector from SAMTEC and
> are connected trough a EEDP-016-12.00-RA1-RA2-2 cross cable from SAMTEC.
> I hope this information is sufficient since there is not much one can
> find about it on Google. In addition, you can see a picture of the board
> including the connector in the datasheet located at
> http://www.windriver.com/products/product-notes/SBC8548E-product-note.pdf.
> It is the connector on the left side of the PCI-EX slot.
>
> We have tried your suggestion but the situation does not change other
> than the lane-mode being set to single lane 0, it still locks up when
> trying to generate a maintenance transaction. I still think it is memory
> related since the lock up occurs when accessing the maintenance window.
> Although all memory related settings seems to be alright.
>
> The kernel output is as follows:
>
> Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
> fsl-of-rio e00c0000.rapidio: Of-device full name
> /soc8548@e0000000/rapidio@c0000
> fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
> fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
> 0x0000000010000000.
> fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
> fsl-of-rio e00c0000.rapidio: DeviceID is 0x0
> fsl-of-rio e00c0000.rapidio: Configured as HOST
> fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
> fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
> fsl-of-rio e00c0000.rapidio: Hardware port width: 4
> fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
> fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
> fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO
> Maintainance Window Size 0x400000,New Main Start: 0xd1080000
> RIO: enumerate master port 0, RIO0 mport
> fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
> fsl_rio_config_read: Passed IS_ALIGNED.
> fsl_rio_config_read: Passed 'out_be32_1'
> fsl_rio_config_read: Passed 'out_be32_2'
> fsl_rio_config_read: len is 4
> fsl_rio_config_read: triggering '__fsl_read_rio_config'
> fsl_rio_config_read: going to request to read data at d108006
>
> Regards,
> Bastiaan
>
> 2010/10/4 Bounine, Alexandre <Alexandre.Bounine@idt.com
> <mailto:Alexandre.Bounine@idt.com>>
>
>     Hi Bastiaan,
>
>     Are you trying board-to-board connection?
>     I am not familiar with WRS SBC8548 board - which type of connector they
>     use for SRIO?
>
>     Assuming that all configuration is correct,
>     I would recommend first to try setting up x1 link mode at the lowest
>     link speed.
>     The x4 mode may present challenges in some cases.
>
>     For quick test you may just add port width override into fsl_rio.c
>     like shown below (ugly but sometimes it helps ;) ):
>
>     @@ -1461,10 +1461,16 @@ int fsl_rio_setup(struct platform_device *dev)
>             rio_register_mport(port);
>
>             priv->regs_win = ioremap(regs.start, regs.end - regs.start + 1);
>             rio_regs_win = priv->regs_win;
>
>     +dev_info(&dev->dev, "Overriding RIO_PORT setting to single lane 0\n");
>     +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
>     0x800000);
>     +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
>     0x2000000);
>     +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) &
>     ~0x800000);
>     +msleep(100);
>     +
>             /* Probe the master port phy type */
>             ccsr = in_be32(priv->regs_win + RIO_CCSR);
>             port->phy_type = (ccsr & 1) ? RIO_PHY_SERIAL : RIO_PHY_PARALLEL;
>             dev_info(&dev->dev, "RapidIO PHY type: %s\n",
>                             (port->phy_type == RIO_PHY_PARALLEL) ?
>     "parallel" :
>
>
>     Let me know what happens.
>     Please keep me in the CC: list next time when posting RapidIO questions
>     to the linuxppc-dev or kernel mailing lists.
>
>     Regards,
>
>     Alex.
>
>
>
>
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

-- 
John Traill
Systems Engineer
Network and Computing Systems Group

Freescale Semiconductor UK LTD
Colvilles Road
East Kilbride
Glasgow G75 0TG, Scotland

Tel: +44 (0) 1355 355494
Fax: +44 (0) 1355 261790

E-mail: john.traill@freescale.com

Registration Number: SC262720
VAT Number: GB831329053

[ ] General Business Use
[ ] Freescale Internal Use Only
[ ] Freescale Confidential Proprietary

Re: Serial RapidIO Maintaintance read causes lock up

[Bastiaan Nijkamp]

[-- Attachment #1: Type: text/plain, Size: 3887 bytes --]

Hi Alex,

Thanks for your advice. We are trying to make a board-to-board connection
without any additional hardware (eg. a switch). The boards use a 50-pin,
right-angle MEC8-125-02-L-D-RA1 connector from SAMTEC and are connected
trough a EEDP-016-12.00-RA1-RA2-2 cross cable from SAMTEC. I hope this
information is sufficient since there is not much one can find about it on
Google. In addition, you can see a picture of the board including the
connector in the datasheet located at
http://www.windriver.com/products/product-notes/SBC8548E-product-note.pdf.
It is the connector on the left side of the PCI-EX slot.

We have tried your suggestion but the situation does not change other than
the lane-mode being set to single lane 0, it still locks up when trying to
generate a maintenance transaction. I still think it is memory related since
the lock up occurs when accessing the maintenance window. Although all
memory related settings seems to be alright.

The kernel output is as follows:

Setting up RapidIO peer-to-peer network /soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Of-device full name
/soc8548@e0000000/rapidio@c0000
fsl-of-rio e00c0000.rapidio: Regs: [mem 0xe00c0000-0xe00dffff]
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, size
0x0000000010000000.
fsl-of-rio e00c0000.rapidio: pwirq: 48, bellirq: 50, txirq: 53, rxirq 54
fsl-of-rio e00c0000.rapidio: DeviceID is 0x0
fsl-of-rio e00c0000.rapidio: Configured as HOST
fsl-of-rio e00c0000.rapidio: Overriding RIO_PORT setting to single lane 0
fsl-of-rio e00c0000.rapidio: RapidIO PHY type: serial
fsl-of-rio e00c0000.rapidio: Hardware port width: 4
fsl-of-rio e00c0000.rapidio: Training connection status: Single-lane 0
fsl-of-rio e00c0000.rapidio: RapidIO Common Transport System size: 256
fsl-of-rio e00c0000.rapidio: LAW start 0x00000000c0000000, RIO Maintainance
Window Size 0x400000,New Main Start: 0xd1080000
RIO: enumerate master port 0, RIO0 mport
fsl_rio_config_read: index 0 destid 255 hopcount 0 offset 00000068 len 4
fsl_rio_config_read: Passed IS_ALIGNED.
fsl_rio_config_read: Passed 'out_be32_1'
fsl_rio_config_read: Passed 'out_be32_2'
fsl_rio_config_read: len is 4
fsl_rio_config_read: triggering '__fsl_read_rio_config'
fsl_rio_config_read: going to request to read data at d108006

Regards,
Bastiaan
2010/10/4 Bounine, Alexandre <Alexandre.Bounine@idt.com>

> Hi Bastiaan,
>
> Are you trying board-to-board connection?
> I am not familiar with WRS SBC8548 board - which type of connector they
> use for SRIO?
>
> Assuming that all configuration is correct,
> I would recommend first to try setting up x1 link mode at the lowest
> link speed.
> The x4 mode may present challenges in some cases.
>
> For quick test you may just add port width override into fsl_rio.c
> like shown below (ugly but sometimes it helps ;) ):
>
> @@ -1461,10 +1461,16 @@ int fsl_rio_setup(struct platform_device *dev)
>        rio_register_mport(port);
>
>        priv->regs_win = ioremap(regs.start, regs.end - regs.start + 1);
>        rio_regs_win = priv->regs_win;
>
> +dev_info(&dev->dev, "Overriding RIO_PORT setting to single lane 0\n");
> +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
> 0x800000);
> +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
> 0x2000000);
> +out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) &
> ~0x800000);
> +msleep(100);
> +
>        /* Probe the master port phy type */
>        ccsr = in_be32(priv->regs_win + RIO_CCSR);
>        port->phy_type = (ccsr & 1) ? RIO_PHY_SERIAL : RIO_PHY_PARALLEL;
>        dev_info(&dev->dev, "RapidIO PHY type: %s\n",
>                        (port->phy_type == RIO_PHY_PARALLEL) ?
> "parallel" :
>
>
> Let me know what happens.
> Please keep me in the CC: list next time when posting RapidIO questions
> to the linuxppc-dev or kernel mailing lists.
>
> Regards,
>
> Alex.
>
>

[-- Attachment #2: Type: text/html, Size: 4562 bytes --]

Re: Serial RapidIO Maintaintance read causes lock up

[Bounine, Alexandre]

Hi Bastiaan,

Are you trying board-to-board connection?
I am not familiar with WRS SBC8548 board - which type of connector they
use for SRIO?

Assuming that all configuration is correct,
I would recommend first to try setting up x1 link mode at the lowest
link speed.
The x4 mode may present challenges in some cases.

For quick test you may just add port width override into fsl_rio.c
like shown below (ugly but sometimes it helps ;) ):

@@ -1461,10 +1461,16 @@ int fsl_rio_setup(struct platform_device *dev)
 	rio_register_mport(port);
=20
 	priv->regs_win =3D ioremap(regs.start, regs.end - regs.start + 1);
 	rio_regs_win =3D priv->regs_win;
=20
+dev_info(&dev->dev, "Overriding RIO_PORT setting to single lane 0\n");
+out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
0x800000);
+out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) |
0x2000000);
+out_be32(priv->regs_win + 0x15C, in_be32(priv->regs_win + 0x15C) &
~0x800000);
+msleep(100);
+
 	/* Probe the master port phy type */
 	ccsr =3D in_be32(priv->regs_win + RIO_CCSR);
 	port->phy_type =3D (ccsr & 1) ? RIO_PHY_SERIAL : RIO_PHY_PARALLEL;
 	dev_info(&dev->dev, "RapidIO PHY type: %s\n",
 			(port->phy_type =3D=3D RIO_PHY_PARALLEL) ?
"parallel" :


Let me know what happens.
Please keep me in the CC: list next time when posting RapidIO questions
to the linuxppc-dev or kernel mailing lists.

Regards,

Alex.
=20