Re: Freescale P2020 CPU Freeze over PCIe abort signal

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[siva kumar]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

Hi ,

Is there any update on this , am getting in to the same state .
https://lists.ozlabs.org/pipermail/linuxppc-dev/2010-October/086680.html

Thanks,
Sivakumar

[-- Attachment #2: Type: text/html, Size: 321 bytes --]

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Scott Wood]

On 01/23/2013 11:41:26 AM, siva kumar wrote:
> Hi ,
>=20
> Is there any update on this , am getting in to the same state .
> https://lists.ozlabs.org/pipermail/linuxppc-dev/2010-October/086680.html

Eran's initial comment of "This should probably go to the Freescale =20
support" seems right.  You can reach Freescale support at =20
support@freescale.com.

-Scott=

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[siva kumar]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

Thank you Scott for the reply.
May I know did u got out of this issue and   can i get some brief on what
changes they had suggested.

Thanks,
Siva

On Thu, Jan 24, 2013 at 3:10 AM, Scott Wood <scottwood@freescale.com> wrote:

> On 01/23/2013 11:41:26 AM, siva kumar wrote:
>
>> Hi ,
>>
>> Is there any update on this , am getting in to the same state .
>> https://lists.ozlabs.org/**pipermail/linuxppc-dev/2010-**
>> October/086680.html<https://lists.ozlabs.org/pipermail/linuxppc-dev/2010-October/086680.html>
>>
>
> Eran's initial comment of "This should probably go to the Freescale
> support" seems right.  You can reach Freescale support at
> support@freescale.com.
>
> -Scott
>

[-- Attachment #2: Type: text/html, Size: 1303 bytes --]

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Scott Wood]

On 01/24/2013 05:53:51 AM, siva kumar wrote:
> Thank you Scott for the reply.
> May I know did u got out of this issue and   can i get some brief on =20
> what
> changes they had suggested.

I wasn't the one that had the issue.  Why not send an e-mail to Eran =20
Liberty?

-Scott

> On Thu, Jan 24, 2013 at 3:10 AM, Scott Wood <scottwood@freescale.com> =20
> wrote:
>=20
> > On 01/23/2013 11:41:26 AM, siva kumar wrote:
> >
> >> Hi ,
> >>
> >> Is there any update on this , am getting in to the same state .
> >> https://lists.ozlabs.org/**pipermail/linuxppc-dev/2010-**
> >> =20
> October/086680.html<https://lists.ozlabs.org/pipermail/linuxppc-dev/2010-=
October/086680.html>
> >>
> >
> > Eran's initial comment of "This should probably go to the Freescale
> > support" seems right.  You can reach Freescale support at
> > support@freescale.com.
> >
> > -Scott
> >
>=20

=

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

Eran Liberty wrote:
> Eran Liberty wrote:
>> This should probably go to the Freescale support, as it feels like a 
>> hardware issue yet the end result is a very frozen Linux kernel so I 
>> post here first...
>>
>> I have a programmable FPGA PCIe device connected to a Freescale's 
>> P2020 PCIe port. As part of the bring-up tests, we are testing two 
>> faulty scenarios:
>> 1. The FPGA totally ignores the PCIe transaction.
>> 2. The FPGA return a transaction abort.
>>
>> Both are plausible PCIe behavior and their should be outcome is 
>> documented in the PCIe spec. The first should be terminated by the 
>> transaction requestor timeout mechanism and raise an error, the 
>> second should abort the transaction and raise and error.
>>
>> In P2020 if I do any of those the CPU is left hung over the transaction.
>>
>> something like:
>> in_le32(addr)
>>
>> is turned into:
>> 7c 00 04 ac     sync   7c 00 4c 2c     lwbrx   r0,0,r9
>> 0c 00 00 00     twi     0,r0,0
>> 4c 00 01 2c     isync
>>
>> assembly code, where in r9 (in this example) hold an address which is 
>> physically mapped into the PCIe resource space.
>>
>> The CPU will hang over the load instruction.
>>
>> Just for the fun of it, I have wrote my own assembly function 
>> omitting everything but the load instruction; still freeze.
>> Replace "lwbrx" with a simple "lwz"; still freeze.
>>
>> It looks like the CPU snoozes till the PCIe transaction is done with 
>> no timeouts, ignoring any abort signal.
>>
>> I am going to:
>> A. Try to reach the Freescale support.
>> B. Asked the FPGA designed to give me a new behavior that will stall 
>> the PCIe transaction replay for 10 sec, but after those return ok.
>> C. report back here with either A or B.
>>
>> If you have any ideas I would love to hear them.
>>
>> -- Liberty
>>
> Some more info:
>
> As said the the FPGA designer provided me a PCIe device that will 
> stall its response to a variable amount of time. The CPU became 
> un-frozen after this amount of time. More over, we have found that in 
> that period till it un-froze the PCIe core did a retry to that 
> transaction over and over every 40 ms. This gave me the bright idea to 
> look for the word "retry" in the Freescale documentation which 
> rewarded me with these registers:
>
> ------------------------------------------------------- snip 
> -------------------------------------------------------
> 16.3.2.3        PCI Express Outbound Completion Timeout Register
>                (PEX_OTB_CPL_TOR)
> The PCI Express outbound completion timeout register, shown in Figure 
> 16-4, contains the maximum wait
> time for a response to come back as a result of an outbound non-posted 
> request before a timeout condition
> occurs.
> Offset 
> 0x00C                                                                                                
> Access: Read/Write
>         0   1              5     7   
> 8                                                                                      
> 31
>     R
>        TD            
> —                                                            TC
>     W
> Reset 0     0  0  0   0   0   0  0   0   0   0   1    0  0   0  0    
> 1   1  1    1   1  1   1   1   1  1   1   1  1  1   1  1
>            Figure 16-4. PCI Express Outbound Completion Timeout 
> Register (PEX_OTB_CPL_TOR)
> Table 16-6 describes the PCI Express outbound completion timeout 
> register fields.
>                                 Table 16-6. PEX_OTB_CPL_TOR Field 
> Descriptions
>  Bits     Name                                                     
> Description
>   0        TD     Timeout disable. This bit controls the 
> enabling/disabling of the timeout function.
>                   0 Enable completion timeout
>                   1 Disable completion timeout
>  1–7        —     Reserved
> 8–31       TC     Timeout counter. This is the value that is used to 
> load the response counter of the completion timeout.
>                   One TC unit is 8× the PCI Express controller clock 
> period; that is, one TC unit is 20 ns at 400 MHz, and 30
>                   ns at 266.66 MHz.
>                   The following are examples of timeout periods based 
> on different TC settings:
>                   0x00_0000 Reserved
>                   0x10_FFFF 22.28 ms at 400 MHz controller clock; 
> 33.34 ms at 266.66 MHz controller clock
>                   0xFF_FFFF 335.54 ms at 400 MHz controller clock; 
> 503.31 ms at 266.66 MHz controller clock
>
>
> 16.3.2.4       PCI Express Configuration Retry Timeout Register
>               (PEX_CONF_RTY_TOR)
> The PCI Express configuration retry timeout register, shown in Figure 
> 16-5, contains the maximum time
> period during which retries of configuration transactions which 
> resulted in a CRS response occur.
> Offset 
> 0x010                                                                               
> Access: Read/Write
>         0  1     3   
> 4                                                                                     
> 31
>     R
>        RD     —                                                 TC
>     W
> Reset 0    0  0  0  0   1  0  0  0  0   0  0   0  0  0  0   1  1  1  
> 1  1  1   1   1 1 1   1  1   1 1   1  1
>           Figure 16-5. PCI Express Configuration Retry Timeout 
> Register (PEX_CONF_RTY_TOR)
>                            QorIQ P2020 Integrated Processor Reference 
> Manual, Rev. 0
> 16-12                                                                                   
> Freescale Semiconductor
>                                                                                                 
> PCI Express Interface Controller
> Table 16-7 describes the PCI Express configuration retry timeout 
> register fields.
>                            Table 16-7. PEX_CONF_RTY_TOR Field 
> Descriptions
>  Bits  Name                                                     
> Description
>   0     RD    Retry disable. This bit disables the retry of a 
> configuration transaction that receives a CRS status response
>               packet.
>               0 Enable retry of a configuration transaction in 
> response to receiving a CRS status response until the timeout
>                  counter (defined by the PEX_CONF_RTY_TOR[TC] field) 
> has expired.
>               1 Disable retry of a configuration transaction 
> regardless of receiving a CRS status response.
>  1–3     —    Reserved
> 4–31    TC    Timeout counter. This is the value that is used to load 
> the CRS response counter.
>               One TC unit is 8× the PCI Express controller clock 
> period; that is, one TC unit is 20 ns at 400 MHz and 30 ns
>               at 266.66 MHz.
>               Timeout period based on different TC settings:
>               0x000_0000        Reserved
>               0x400_FFFF        1.34 s at 400 MHz controller clock, 
> 2.02 s at 266.66 MHz controller clock
>               0xFFF_FFFF        5.37 s at 400 MHz controller clock, 
> 8.05 s at 266.66 MHz controller clock
> ------------------------------------------------------- snap 
> -------------------------------------------------------
>
> Now this is all nice on the paper, but what the P2020 seems to be 
> doing in reality is
> 1. never expire
> 2. do re-tries even in the non configuration access
>
> I am going to try to disable completion timeout and see if I get 
> better behavior.
>
> -- Liberty
>
>
Disabling PEX_OTB_CPL_TOR,  PEX_CONF_RTY_TOR, or both yields the same 
behavior. The kernel freezes over the load command while the underlying 
hardware does PCIe transaction retries to infinity and beyond.

-- Liberty

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

Eran Liberty wrote:
> This should probably go to the Freescale support, as it feels like a 
> hardware issue yet the end result is a very frozen Linux kernel so I 
> post here first...
>
> I have a programmable FPGA PCIe device connected to a Freescale's 
> P2020 PCIe port. As part of the bring-up tests, we are testing two 
> faulty scenarios:
> 1. The FPGA totally ignores the PCIe transaction.
> 2. The FPGA return a transaction abort.
>
> Both are plausible PCIe behavior and their should be outcome is 
> documented in the PCIe spec. The first should be terminated by the 
> transaction requestor timeout mechanism and raise an error, the second 
> should abort the transaction and raise and error.
>
> In P2020 if I do any of those the CPU is left hung over the transaction.
>
> something like:
> in_le32(addr)
>
> is turned into:
> 7c 00 04 ac     sync   7c 00 4c 2c     lwbrx   r0,0,r9
> 0c 00 00 00     twi     0,r0,0
> 4c 00 01 2c     isync
>
> assembly code, where in r9 (in this example) hold an address which is 
> physically mapped into the PCIe resource space.
>
> The CPU will hang over the load instruction.
>
> Just for the fun of it, I have wrote my own assembly function omitting 
> everything but the load instruction; still freeze.
> Replace "lwbrx" with a simple "lwz"; still freeze.
>
> It looks like the CPU snoozes till the PCIe transaction is done with 
> no timeouts, ignoring any abort signal.
>
> I am going to:
> A. Try to reach the Freescale support.
> B. Asked the FPGA designed to give me a new behavior that will stall 
> the PCIe transaction replay for 10 sec, but after those return ok.
> C. report back here with either A or B.
>
> If you have any ideas I would love to hear them.
>
> -- Liberty
>
Some more info:

As said the the FPGA designer provided me a PCIe device that will stall 
its response to a variable amount of time. The CPU became un-frozen 
after this amount of time. More over, we have found that in that period 
till it un-froze the PCIe core did a retry to that transaction over and 
over every 40 ms. This gave me the bright idea to look for the word 
"retry" in the Freescale documentation which rewarded me with these 
registers:

------------------------------------------------------- snip 
-------------------------------------------------------
16.3.2.3        PCI Express Outbound Completion Timeout Register
                (PEX_OTB_CPL_TOR)
The PCI Express outbound completion timeout register, shown in Figure 
16-4, contains the maximum wait
time for a response to come back as a result of an outbound non-posted 
request before a timeout condition
occurs.
 Offset 
0x00C                                                                                                
Access: Read/Write
         0   1              5     7   
8                                                                                      
31
     R
        TD            
—                                                            TC
     W
 Reset 0     0  0  0   0   0   0  0   0   0   0   1    0  0   0  0    
1   1  1    1   1  1   1   1   1  1   1   1  1  1   1  1
            Figure 16-4. PCI Express Outbound Completion Timeout 
Register (PEX_OTB_CPL_TOR)
Table 16-6 describes the PCI Express outbound completion timeout 
register fields.
                                 Table 16-6. PEX_OTB_CPL_TOR Field 
Descriptions
  Bits     Name                                                     
Description
   0        TD     Timeout disable. This bit controls the 
enabling/disabling of the timeout function.
                   0 Enable completion timeout
                   1 Disable completion timeout
  1–7        —     Reserved
 8–31       TC     Timeout counter. This is the value that is used to 
load the response counter of the completion timeout.
                   One TC unit is 8× the PCI Express controller clock 
period; that is, one TC unit is 20 ns at 400 MHz, and 30
                   ns at 266.66 MHz.
                   The following are examples of timeout periods based 
on different TC settings:
                   0x00_0000 Reserved
                   0x10_FFFF 22.28 ms at 400 MHz controller clock; 33.34 
ms at 266.66 MHz controller clock
                   0xFF_FFFF 335.54 ms at 400 MHz controller clock; 
503.31 ms at 266.66 MHz controller clock


16.3.2.4       PCI Express Configuration Retry Timeout Register
               (PEX_CONF_RTY_TOR)
The PCI Express configuration retry timeout register, shown in Figure 
16-5, contains the maximum time
period during which retries of configuration transactions which resulted 
in a CRS response occur.
 Offset 
0x010                                                                               
Access: Read/Write
         0  1     3   
4                                                                                     
31
     R
        RD     —                                                 TC
     W
 Reset 0    0  0  0  0   1  0  0  0  0   0  0   0  0  0  0   1  1  1  1  
1  1   1   1 1 1   1  1   1 1   1  1
           Figure 16-5. PCI Express Configuration Retry Timeout Register 
(PEX_CONF_RTY_TOR)
                            QorIQ P2020 Integrated Processor Reference 
Manual, Rev. 0
16-12                                                                                   
Freescale Semiconductor
                                                                                                 
PCI Express Interface Controller
Table 16-7 describes the PCI Express configuration retry timeout 
register fields.
                            Table 16-7. PEX_CONF_RTY_TOR Field Descriptions
  Bits  Name                                                     Description
   0     RD    Retry disable. This bit disables the retry of a 
configuration transaction that receives a CRS status response
               packet.
               0 Enable retry of a configuration transaction in response 
to receiving a CRS status response until the timeout
                  counter (defined by the PEX_CONF_RTY_TOR[TC] field) 
has expired.
               1 Disable retry of a configuration transaction regardless 
of receiving a CRS status response.
  1–3     —    Reserved
 4–31    TC    Timeout counter. This is the value that is used to load 
the CRS response counter.
               One TC unit is 8× the PCI Express controller clock 
period; that is, one TC unit is 20 ns at 400 MHz and 30 ns
               at 266.66 MHz.
               Timeout period based on different TC settings:
               0x000_0000        Reserved
               0x400_FFFF        1.34 s at 400 MHz controller clock, 
2.02 s at 266.66 MHz controller clock
               0xFFF_FFFF        5.37 s at 400 MHz controller clock, 
8.05 s at 266.66 MHz controller clock
------------------------------------------------------- snap 
-------------------------------------------------------

Now this is all nice on the paper, but what the P2020 seems to be doing 
in reality is
1. never expire
2. do re-tries even in the non configuration access

I am going to try to disable completion timeout and see if I get better 
behavior.

-- Liberty

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

tiejun.chen wrote:
> AFAIK we can set one bit on PEX_ERR_DISR to detect PCI Express completion
> time-out. If so one interrupt should be issued. But I'm not sure if this can fix
> your issue.
>
> Tiejun
>   
As I understand the problem, this will not help me as the CPU itself is 
on hold waiting for the assembly line to complete. It may give me enough 
to spot the faulty situation and crash the system... but I am aiming for 
an Enterprise grade product.  Crash/oops is not something I want to put 
into the system if I do not absolutely have to and I do have a hardware 
watch-dog that will pull me out if I do.

So just for the fun of it I am going to follow up on this PEX_ERR_DISR, 
but I do not see how it will help.

-- Liberty

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[tiejun.chen]

Eran Liberty wrote:
> This should probably go to the Freescale support, as it feels like a
> hardware issue yet the end result is a very frozen Linux kernel so I
> post here first...
> 
> I have a programmable FPGA PCIe device connected to a Freescale's P2020
> PCIe port. As part of the bring-up tests, we are testing two faulty
> scenarios:
> 1. The FPGA totally ignores the PCIe transaction.
> 2. The FPGA return a transaction abort.
> 
> Both are plausible PCIe behavior and their should be outcome is
> documented in the PCIe spec. The first should be terminated by the
> transaction requestor timeout mechanism and raise an error, the second
> should abort the transaction and raise and error.
> 
> In P2020 if I do any of those the CPU is left hung over the transaction.
> 
> something like:
> in_le32(addr)
> 
> is turned into:
> 7c 00 04 ac     sync   7c 00 4c 2c     lwbrx   r0,0,r9
> 0c 00 00 00     twi     0,r0,0
> 4c 00 01 2c     isync
> 
> assembly code, where in r9 (in this example) hold an address which is
> physically mapped into the PCIe resource space.
> 
> The CPU will hang over the load instruction.
> 
> Just for the fun of it, I have wrote my own assembly function omitting
> everything but the load instruction; still freeze.
> Replace "lwbrx" with a simple "lwz"; still freeze.
> 
> It looks like the CPU snoozes till the PCIe transaction is done with no
> timeouts, ignoring any abort signal.
> 

AFAIK we can set one bit on PEX_ERR_DISR to detect PCI Express completion
time-out. If so one interrupt should be issued. But I'm not sure if this can fix
your issue.

Tiejun

> I am going to:
> A. Try to reach the Freescale support.
> B. Asked the FPGA designed to give me a new behavior that will stall the
> PCIe transaction replay for 10 sec, but after those return ok.
> C. report back here with either A or B.
> 
> If you have any ideas I would love to hear them.
> 
> -- Liberty
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
> 

Re: Freescale P2020 CPU Freeze over PCIe abort signal

[Bin Meng]

On Mon, Oct 18, 2010 at 3:24 AM, Eran Liberty <liberty@extricom.com> wrote:
<snip>

> In P2020 if I do any of those the CPU is left hung over the transaction.
>
> something like:
> in_le32(addr)
>
> is turned into:
> 7c 00 04 ac =A0 =A0 sync =A0 7c 00 4c 2c =A0 =A0 lwbrx =A0 r0,0,r9
> 0c 00 00 00 =A0 =A0 twi =A0 =A0 0,r0,0
> 4c 00 01 2c =A0 =A0 isync
>
> assembly code, where in r9 (in this example) hold an address which is
> physically mapped into the PCIe resource space.
>
> The CPU will hang over the load instruction.
>
> Just for the fun of it, I have wrote my own assembly function omitting
> everything but the load instruction; still freeze.
> Replace "lwbrx" with a simple "lwz"; still freeze.
>
> It looks like the CPU snoozes till the PCIe transaction is done with no
> timeouts, ignoring any abort signal.
>

It sounds like a similar issue I got with the 83xx PCIe host controller.

If there is no valid link established under the PCIe host controller
(ie: no device connected), or the device under the PCIe host
controller does not respond the configuration access correctly,
the host will hang at the instruction of "lwz" when trying to access
the PCIe configuration space (in 83xx, it's memory mapped)

And I remember the same issue exists in 8548, not sure if it is fixed in P2=
020.

<snip>

Bin

> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
>

Freescale P2020 CPU Freeze over PCIe abort signal

[Eran Liberty]

This should probably go to the Freescale support, as it feels like a 
hardware issue yet the end result is a very frozen Linux kernel so I 
post here first...

I have a programmable FPGA PCIe device connected to a Freescale's P2020 
PCIe port. As part of the bring-up tests, we are testing two faulty 
scenarios:
1. The FPGA totally ignores the PCIe transaction.
2. The FPGA return a transaction abort.

Both are plausible PCIe behavior and their should be outcome is 
documented in the PCIe spec. The first should be terminated by the 
transaction requestor timeout mechanism and raise an error, the second 
should abort the transaction and raise and error.

In P2020 if I do any of those the CPU is left hung over the transaction.

something like:
in_le32(addr)

is turned into:
7c 00 04 ac     sync   
7c 00 4c 2c     lwbrx   r0,0,r9
0c 00 00 00     twi     0,r0,0
4c 00 01 2c     isync

assembly code, where in r9 (in this example) hold an address which is 
physically mapped into the PCIe resource space.

The CPU will hang over the load instruction.

Just for the fun of it, I have wrote my own assembly function omitting 
everything but the load instruction; still freeze.
Replace "lwbrx" with a simple "lwz"; still freeze.

It looks like the CPU snoozes till the PCIe transaction is done with no 
timeouts, ignoring any abort signal.

I am going to:
A. Try to reach the Freescale support.
B. Asked the FPGA designed to give me a new behavior that will stall the 
PCIe transaction replay for 10 sec, but after those return ok.
C. report back here with either A or B.

If you have any ideas I would love to hear them.

-- Liberty