Bug in p1_p2_rdb_pc? Caching-inhibited bit for initial L2 SRAM entry in TLB

Bug in p1_p2_rdb_pc? Caching-inhibited bit for initial L2 SRAM entry in TLB

[Pali Rohár]

Hello!

I suspect that there is a bug in board/freescale/p1_p2_rdb_pc/tlb.c code
which configures TLB entry for initial L2 SRAM.

When L2 is 512 kB long (e.g. on P2020) then U-Boot *unsets* MAS2_I bit
for first half of L2 and for second half of L2 U-Boot *sets* this bit.

See code:
https://source.denx.de/u-boot/u-boot/-/blob/v2022.04/board/freescale/p1_p2_rdb_pc/tlb.c#L99-104

I do not think that one part of L2 SRAM should be configured differently
as second part. Therefore I think that this is a bug in U-Boot code.

Do you know is correct configuration of TLB entries for initial L2 SRAM?

MAS2_I is Caching-inhibited bit which is described as:

Caching-inhibited:
* 0 - Accesses to this page are considered cacheable.
* 1 - The page is considered caching-inhibited. All loads and stores to
      the page bypass the caches and are performed directly to main
      memory. A read or write to a caching-inhibited page affects only
      the memory element specified by the operation.

Re: Bug in p1_p2_rdb_pc? Caching-inhibited bit for initial L2 SRAM entry in TLB

[Pali Rohár]

On Tuesday 05 April 2022 10:57:37 Pali Rohár wrote:
> Hello!
> 
> I suspect that there is a bug in board/freescale/p1_p2_rdb_pc/tlb.c code
> which configures TLB entry for initial L2 SRAM.
> 
> When L2 is 512 kB long (e.g. on P2020) then U-Boot *unsets* MAS2_I bit
> for first half of L2 and for second half of L2 U-Boot *sets* this bit.
> 
> See code:
> https://source.denx.de/u-boot/u-boot/-/blob/v2022.04/board/freescale/p1_p2_rdb_pc/tlb.c#L99-104
> 
> I do not think that one part of L2 SRAM should be configured differently
> as second part. Therefore I think that this is a bug in U-Boot code.
> 
> Do you know is correct configuration of TLB entries for initial L2 SRAM?
> 
> MAS2_I is Caching-inhibited bit which is described as:
> 
> Caching-inhibited:
> * 0 - Accesses to this page are considered cacheable.
> * 1 - The page is considered caching-inhibited. All loads and stores to
>       the page bypass the caches and are performed directly to main
>       memory. A read or write to a caching-inhibited page affects only
>       the memory element specified by the operation.

Hello! I found EREF: A Programmer’s Reference Manual for Freescale Power
Architecture Processors Supports e500 core family (e500v1, e500v2,
e500mc, e5500, e6500) e200 core family document at NXP web:

https://www.nxp.com/files-static/32bit/doc/ref_manual/EREF_RM.pdf

And section "Cache and MMU Architecture" in part 7.3.1.2.2 Unable to
Lock Conditions (page 763) contains following information:

If no exceptions occur and no overlocking condition exists, an attempt
to set a lock can fail if any of the following is true:

• The target address is marked cache-inhibited, or the storage
  attributes of the address uses a coherency protocol that does not
  support locking

So for me it looks like that L2 SRAM (which works at L2 with locked
cache lines) should not set MA2_I (cache-inhibited) bit.

Any opinion? Or you do have some more information?

Re: Bug in p1_p2_rdb_pc? Caching-inhibited bit for initial L2 SRAM entry in TLB

[Pali Rohár]

+ Sinan

On Wednesday 13 April 2022 11:26:33 Pali Rohár wrote:
> On Tuesday 05 April 2022 10:57:37 Pali Rohár wrote:
> > Hello!
> > 
> > I suspect that there is a bug in board/freescale/p1_p2_rdb_pc/tlb.c code
> > which configures TLB entry for initial L2 SRAM.
> > 
> > When L2 is 512 kB long (e.g. on P2020) then U-Boot *unsets* MAS2_I bit
> > for first half of L2 and for second half of L2 U-Boot *sets* this bit.
> > 
> > See code:
> > https://source.denx.de/u-boot/u-boot/-/blob/v2022.04/board/freescale/p1_p2_rdb_pc/tlb.c#L99-104
> > 
> > I do not think that one part of L2 SRAM should be configured differently
> > as second part. Therefore I think that this is a bug in U-Boot code.
> > 
> > Do you know is correct configuration of TLB entries for initial L2 SRAM?
> > 
> > MAS2_I is Caching-inhibited bit which is described as:
> > 
> > Caching-inhibited:
> > * 0 - Accesses to this page are considered cacheable.
> > * 1 - The page is considered caching-inhibited. All loads and stores to
> >       the page bypass the caches and are performed directly to main
> >       memory. A read or write to a caching-inhibited page affects only
> >       the memory element specified by the operation.
> 
> Hello! I found EREF: A Programmer’s Reference Manual for Freescale Power
> Architecture Processors Supports e500 core family (e500v1, e500v2,
> e500mc, e5500, e6500) e200 core family document at NXP web:
> 
> https://www.nxp.com/files-static/32bit/doc/ref_manual/EREF_RM.pdf
> 
> And section "Cache and MMU Architecture" in part 7.3.1.2.2 Unable to
> Lock Conditions (page 763) contains following information:
> 
> If no exceptions occur and no overlocking condition exists, an attempt
> to set a lock can fail if any of the following is true:
> 
> • The target address is marked cache-inhibited, or the storage
>   attributes of the address uses a coherency protocol that does not
>   support locking
> 
> So for me it looks like that L2 SRAM (which works at L2 with locked
> cache lines) should not set MA2_I (cache-inhibited) bit.
> 
> Any opinion? Or you do have some more information?

Re: Bug in p1_p2_rdb_pc? Caching-inhibited bit for initial L2 SRAM entry in TLB

[Pali Rohár]

On Thursday 14 April 2022 23:05:39 Pali Rohár wrote:
> + Sinan
> 
> On Wednesday 13 April 2022 11:26:33 Pali Rohár wrote:
> > On Tuesday 05 April 2022 10:57:37 Pali Rohár wrote:
> > > Hello!
> > > 
> > > I suspect that there is a bug in board/freescale/p1_p2_rdb_pc/tlb.c code
> > > which configures TLB entry for initial L2 SRAM.
> > > 
> > > When L2 is 512 kB long (e.g. on P2020) then U-Boot *unsets* MAS2_I bit
> > > for first half of L2 and for second half of L2 U-Boot *sets* this bit.
> > > 
> > > See code:
> > > https://source.denx.de/u-boot/u-boot/-/blob/v2022.04/board/freescale/p1_p2_rdb_pc/tlb.c#L99-104
> > > 
> > > I do not think that one part of L2 SRAM should be configured differently
> > > as second part. Therefore I think that this is a bug in U-Boot code.
> > > 
> > > Do you know is correct configuration of TLB entries for initial L2 SRAM?
> > > 
> > > MAS2_I is Caching-inhibited bit which is described as:
> > > 
> > > Caching-inhibited:
> > > * 0 - Accesses to this page are considered cacheable.
> > > * 1 - The page is considered caching-inhibited. All loads and stores to
> > >       the page bypass the caches and are performed directly to main
> > >       memory. A read or write to a caching-inhibited page affects only
> > >       the memory element specified by the operation.
> > 
> > Hello! I found EREF: A Programmer’s Reference Manual for Freescale Power
> > Architecture Processors Supports e500 core family (e500v1, e500v2,
> > e500mc, e5500, e6500) e200 core family document at NXP web:
> > 
> > https://www.nxp.com/files-static/32bit/doc/ref_manual/EREF_RM.pdf
> > 
> > And section "Cache and MMU Architecture" in part 7.3.1.2.2 Unable to
> > Lock Conditions (page 763) contains following information:
> > 
> > If no exceptions occur and no overlocking condition exists, an attempt
> > to set a lock can fail if any of the following is true:
> > 
> > • The target address is marked cache-inhibited, or the storage
> >   attributes of the address uses a coherency protocol that does not
> >   support locking
> > 
> > So for me it looks like that L2 SRAM (which works at L2 with locked
> > cache lines) should not set MA2_I (cache-inhibited) bit.
> > 
> > Any opinion? Or you do have some more information?

Hello!

I looked at it again and it is more complicated as I initially thought.

There are two options how L2 cache on P2020 may be used as SRAM. First
option is the classic way with locking lines, like it is done on other
architectures. Second option seems to be P1/P2 specific as it is *not*
documented in e500 core reference manual, but in P2020 SoC reference
manual, and this option changes L2 operation from Cache to Memory-Mapped
SRAM mode.

Downloading P2020 reference manual (rev2) requires NXP account:
https://www.nxp.com/webapp/Download?colCode=P2020RM

But some older version (rev0) can be found on internet, e.g.:
http://m4udit.dinauz.org/P2020RM_rev0.pdf

I checked U-Boot code and it is for L2 SRAM configuration uses second
option, therefore not via L2 locked lines, but as L2 memory-mapping.

P2020 reference manual in section "Memory-Mapped SRAM Coherency Rules"
contains:

  Accesses to memory-mapped SRAM are cacheable only in the corresponding
  e500 L1 caches. External accesses must be marked cache-inhibited or be
  performed with non-caching transactions.

So based on the fact that L2 is in U-Boot in memory-mapped SRAM mode,
not in cache mode with locked lines and that P2020 RM explicitly says
that memory-mapped SRAM can be cacheable in L1, my understanding is that
TLB mapping for L2 SRAM should work with Caching-inhibited bit set and
also with unset (when unset then caching is disabled at L1 level).

Is my deduction correct?

Priyanka, Sinan, any idea?

Re: Bug in p1_p2_rdb_pc? Caching-inhibited bit for initial L2 SRAM entry in TLB

[Sinan Akman]

   Hi Pali

On 2022-05-08 11:08 a.m., Pali Rohár wrote:
> On Thursday 14 April 2022 23:05:39 Pali Rohár wrote:
>> + Sinan
>>
>> On Wednesday 13 April 2022 11:26:33 Pali Rohár wrote:
>>> On Tuesday 05 April 2022 10:57:37 Pali Rohár wrote:
>>>> Hello!
>>>>
>>>> I suspect that there is a bug in board/freescale/p1_p2_rdb_pc/tlb.c code
>>>> which configures TLB entry for initial L2 SRAM.
>>>>
>>>> When L2 is 512 kB long (e.g. on P2020) then U-Boot *unsets* MAS2_I bit
>>>> for first half of L2 and for second half of L2 U-Boot *sets* this bit.
>>>>
>>>> See code:
>>>> https://source.denx.de/u-boot/u-boot/-/blob/v2022.04/board/freescale/p1_p2_rdb_pc/tlb.c#L99-104
>>>>
>>>> I do not think that one part of L2 SRAM should be configured differently
>>>> as second part. Therefore I think that this is a bug in U-Boot code.
>>>>
>>>> Do you know is correct configuration of TLB entries for initial L2 SRAM?
>>>>
>>>> MAS2_I is Caching-inhibited bit which is described as:
>>>>
>>>> Caching-inhibited:
>>>> * 0 - Accesses to this page are considered cacheable.
>>>> * 1 - The page is considered caching-inhibited. All loads and stores to
>>>>        the page bypass the caches and are performed directly to main
>>>>        memory. A read or write to a caching-inhibited page affects only
>>>>        the memory element specified by the operation.
>>> Hello! I found EREF: A Programmer’s Reference Manual for Freescale Power
>>> Architecture Processors Supports e500 core family (e500v1, e500v2,
>>> e500mc, e5500, e6500) e200 core family document at NXP web:
>>>
>>> https://www.nxp.com/files-static/32bit/doc/ref_manual/EREF_RM.pdf
>>>
>>> And section "Cache and MMU Architecture" in part 7.3.1.2.2 Unable to
>>> Lock Conditions (page 763) contains following information:
>>>
>>> If no exceptions occur and no overlocking condition exists, an attempt
>>> to set a lock can fail if any of the following is true:
>>>
>>> • The target address is marked cache-inhibited, or the storage
>>>    attributes of the address uses a coherency protocol that does not
>>>    support locking
>>>
>>> So for me it looks like that L2 SRAM (which works at L2 with locked
>>> cache lines) should not set MA2_I (cache-inhibited) bit.
>>>
>>> Any opinion? Or you do have some more information?
> Hello!
>
> I looked at it again and it is more complicated as I initially thought.
>
> There are two options how L2 cache on P2020 may be used as SRAM. First
> option is the classic way with locking lines, like it is done on other
> architectures. Second option seems to be P1/P2 specific as it is *not*
> documented in e500 core reference manual, but in P2020 SoC reference
> manual, and this option changes L2 operation from Cache to Memory-Mapped
> SRAM mode.
>
> Downloading P2020 reference manual (rev2) requires NXP account:
> https://www.nxp.com/webapp/Download?colCode=P2020RM
>
> But some older version (rev0) can be found on internet, e.g.:
> http://m4udit.dinauz.org/P2020RM_rev0.pdf
>
> I checked U-Boot code and it is for L2 SRAM configuration uses second
> option, therefore not via L2 locked lines, but as L2 memory-mapping.
>
> P2020 reference manual in section "Memory-Mapped SRAM Coherency Rules"
> contains:
>
>    Accesses to memory-mapped SRAM are cacheable only in the corresponding
>    e500 L1 caches. External accesses must be marked cache-inhibited or be
>    performed with non-caching transactions.
>
> So based on the fact that L2 is in U-Boot in memory-mapped SRAM mode,
> not in cache mode with locked lines and that P2020 RM explicitly says
> that memory-mapped SRAM can be cacheable in L1, my understanding is that
> TLB mapping for L2 SRAM should work with Caching-inhibited bit set and
> also with unset (when unset then caching is disabled at L1 level).
>
> Is my deduction correct?
>
> Priyanka, Sinan, any idea?

   I am still away from my boards but in few weeks I will
take a deeper look at this. I need to run some test to see
and understand how this is all implemented.

   Thanks for all the work you do for P2020.

   Regards
   Sinan Akman