kexec on e300 core / mpc5121

kexec on e300 core / mpc5121

[Sebastian Andrzej Siewior]

I've tried kexec on e300 core which should be easy since it is possible
to disable the MMU on that core. However it does not work.
Once I disable the MMU, I can't access my MBAR and print chars on the
serial port. Is this "normal" or do I have still some caches on?
After I've setup two BATs one for SDRAM and the other for MBAR and
enabled the MMU again, I could trace it and see that it jumped into the
new kernel. I got lost after the turn off mmu part. So the new kernel
does not boot but it gets there.

I noticed that if I boot from u-boot, my I & Dcache are switched off and
in kexec mode there are on. So I tried to disable them in my kernel
wrapper with the following sequence which I borrowed from u-boot:

|icache_disable:
|        mfspr   r3, HID0
|        lis     r4, 0
|        ori     r4, r4, HID0_ICE|HID0_ILOCK
|        andc    r3, r3, r4
|        ori     r4, r3, HID0_ICFI
|        isync
|        mtspr   HID0, r4     /* sets invalidate, clears enable and lock*/
|        isync
|        mtspr   HID0, r3        /* clears invalidate */
|        blr
|dcache_disable:
|        mfspr   r3, HID0
|        lis     r4, 0
|        ori     r4, r4, HID0_DCE|HID0_DLOCK
|        andc    r3, r3, r4
|        ori     r4, r3, HID0_DCI
|        sync
|        mtspr   HID0, r4        /* sets invalidate, clears enable and lock */
|        sync
|        mtspr   HID0, r3        /* clears invalidate */
|        blr

First icache, followed by dcache.
This has the side effect that my LR which was saved on stack suddenly
become 0x0 in the function where I disable the caches.
So it looks like the sequence above is not correct and it is probably a
cache thing why the kernel does not boot.

So I tried not to enable I & Dcache in first place. The kernel booted
uncached and then I tried to kexec into my kernel. This did not succeed,
I saw my last char printed in kernel code but nothing more. So it looks
like the same result like the cached attempt.

The HID0 register differs only in powermanagement, MSR differs in IR,DR
and PR. Is there another register which could be different? There should
be one, since I can't write anything on my serial line unless I setup
BATs and enable data address translation.
Does someone have an idea?

Sebastian

Re: kexec on e300 core / mpc5121

[Sebastian Andrzej Siewior]

* Sebastian Andrzej Siewior | 2009-08-05 01:06:06 [+0200]:
[long mail]
>Does someone have an idea?
Issue fixed, it was the ipic. The missing data while disabling the
caches was my fault because I invalidated the caches before I've flushed
them.

Sebastian

Re: kexec on e300 core / mpc5121

[Kenneth Johansson]

On Wed, 2009-08-05 at 01:06 +0200, Sebastian Andrzej Siewior wrote:
> I've tried kexec on e300 core which should be easy since it is possible
> to disable the MMU on that core. However it does not work.

Is it not possible to disable the mmu on all cpu's that have one ?? 

> Once I disable the MMU, I can't access my MBAR and print chars on the
> serial port. Is this "normal" or do I have still some caches on?

Yes cache and mmu is separate. the 5121 is not cache coherent and do not
limit cache to only memory regions so serial port or any memory mapped
register is a no no unless you have cache off or cache on and mmu on
with a correct setting for what address range to cache. 

Before you turn off the cache you need to flush out all dirty data. best
done by simply reading in 32kb of crap from somewhere. otherwise you are
sure to loose at least the stack and you do not want that.

Re: kexec on e300 core / mpc5121

[Scott Wood]

On Thu, Aug 06, 2009 at 12:49:45AM +0200, Kenneth Johansson wrote:
> On Wed, 2009-08-05 at 01:06 +0200, Sebastian Andrzej Siewior wrote:
> > I've tried kexec on e300 core which should be easy since it is possible
> > to disable the MMU on that core. However it does not work.
> 
> Is it not possible to disable the mmu on all cpu's that have one ?? 

No, on e500 for example it is always on.  You can use large pages with
identity maps to make it seem like it's off, though.

> Before you turn off the cache you need to flush out all dirty data. best
> done by simply reading in 32kb of crap from somewhere. otherwise you are
> sure to loose at least the stack and you do not want that.

32KiB is usually not sufficient -- depending on the initial state, an
8-way 32KiB cache with PLRU (such as in e300) can require up to 52KiB of
data (13 loads per set) to fully flush if you simply load+dcbf (in
separate passes) an arbitrary chunk of data which may already be in the
cache.

If instead you load+dcbf something that you know is not already in the
cache, or if you have a flush-assist mode that does not choose vacant
cache lines when available (instead sticking strictly to the PLRU), the
maximum is 48KiB.

If you have flush-assist *and* you guarantee no hit on the flush data,
then you can get away with only 32KiB.

-Scott

Re: kexec on e300 core / mpc5121

[Kenneth Johansson]

On Wed, 2009-08-05 at 18:47 -0500, Scott Wood wrote:
> On Thu, Aug 06, 2009 at 12:49:45AM +0200, Kenneth Johansson wrote:
> > On Wed, 2009-08-05 at 01:06 +0200, Sebastian Andrzej Siewior wrote:
> > > I've tried kexec on e300 core which should be easy since it is possible
> > > to disable the MMU on that core. However it does not work.
> > 
> > Is it not possible to disable the mmu on all cpu's that have one ?? 
> 
> No, on e500 for example it is always on.  You can use large pages with
> identity maps to make it seem like it's off, though.

why do something like that ? 

> > Before you turn off the cache you need to flush out all dirty data. best
> > done by simply reading in 32kb of crap from somewhere. otherwise you are
> > sure to loose at least the stack and you do not want that.
> 
> 32KiB is usually not sufficient -- depending on the initial state, an
> 8-way 32KiB cache with PLRU (such as in e300) can require up to 52KiB of
> data (13 loads per set) to fully flush if you simply load+dcbf (in
> separate passes) an arbitrary chunk of data which may already be in the
> cache.

if you have 
int crap[1024*32/4] __attribute__((aligned(32)))

What will happen with the cache if you just load data into a register
from that array ?? Wont it force out everything else in the cache to
make room for the crap ? 

 

Re: kexec on e300 core / mpc5121

[Sebastian Andrzej Siewior]

* Kenneth Johansson | 2009-08-06 02:20:50 [+0200]:

>> > Is it not possible to disable the mmu on all cpu's that have one ?? 
>> 
>> No, on e500 for example it is always on.  You can use large pages with
>> identity maps to make it seem like it's off, though.
>
>why do something like that ? 
on e500 you simply can't disable the mmu. It is always on even if you
come out of reset.

Sebastian

Re: kexec on e300 core / mpc5121

[Kumar Gala]

On Aug 5, 2009, at 5:49 PM, Kenneth Johansson wrote:

> On Wed, 2009-08-05 at 01:06 +0200, Sebastian Andrzej Siewior wrote:
>> I've tried kexec on e300 core which should be easy since it is  
>> possible
>> to disable the MMU on that core. However it does not work.
>
> Is it not possible to disable the mmu on all cpu's that have one ??

Nope.  Book-E class PPC parts always have the mmu.

>> Once I disable the MMU, I can't access my MBAR and print chars on the
>> serial port. Is this "normal" or do I have still some caches on?
>
> Yes cache and mmu is separate. the 5121 is not cache coherent and do  
> not
> limit cache to only memory regions so serial port or any memory mapped
> register is a no no unless you have cache off or cache on and mmu on
> with a correct setting for what address range to cache.
>
> Before you turn off the cache you need to flush out all dirty data.  
> best
> done by simply reading in 32kb of crap from somewhere. otherwise you  
> are
> sure to loose at least the stack and you do not want that.

- k

Re: kexec on e300 core / mpc5121

[Scott Wood]

On Thu, Aug 06, 2009 at 02:20:50AM +0200, Kenneth Johansson wrote:
> > > Before you turn off the cache you need to flush out all dirty data. best
> > > done by simply reading in 32kb of crap from somewhere. otherwise you are
> > > sure to loose at least the stack and you do not want that.
> > 
> > 32KiB is usually not sufficient -- depending on the initial state, an
> > 8-way 32KiB cache with PLRU (such as in e300) can require up to 52KiB of
> > data (13 loads per set) to fully flush if you simply load+dcbf (in
> > separate passes) an arbitrary chunk of data which may already be in the
> > cache.
> 
> if you have 
> int crap[1024*32/4] __attribute__((aligned(32)))
> 
> What will happen with the cache if you just load data into a register
> from that array ?? Wont it force out everything else in the cache to
> make room for the crap ? 

Under certain circumstances, the newly loaded flush data will replace
previous flush data rather than the actual data you're trying to displace
-- especially if some of the "crap" is already in the cache, or if there
were some empty cache lines at the start of the process.  It's not a true
LRU.

If you read 52K, and then dcbf that same 52K, with interrupts disabled
(they can disturb the PLRU), you should be OK.  It doesn't even need to
be dedicated junk, any memory will work.

-Scott