FTrace on MPC8xx

FTrace on MPC8xx

[Stefan Roese]

Hi,

I noticed that ftrace doesn't seem to work on MPC8xx. I'm trying to use it on 
an MPC855T board, but ftrace oopses upon startup:

[    0.028785] ftrace: allocating 10312 entries in 31 pages
[    0.038594] ------------[ cut here ]------------
[    0.038760] WARNING: at kernel/trace/ftrace.c:1014
[    0.038889] Modules linked in:
[    0.039062] NIP: c00666a0 LR: c006782c CTR: 00000001
[    0.039249] REGS: c0381e60 TRAP: 0700   Not tainted  (2.6.38-default+)
[    0.039417] MSR: 00021032 <ME,CE,IR,DR>  CR: 53009393  XER: a000af7f
[    0.039856] TASK = c0360500[0] 'swapper' THREAD: c0380000
[    0.040001] GPR00: 00000001 c0381f10 c0360500 ffffffff c02962c8 00000000 
c02962c4 00000000 
[    0.040487] GPR08: 1de1a3e3 00000000 000258e3 c0381f40 93009399 10091a60 
03fff800 0040055c 
[    0.040977] GPR16: 00000001 00000001 ffffffff 007fff00 03ff9fb0 00000000 
00000000 c0384080 
[    0.041459] GPR24: c000f234 00000000 024acd00 00009032 c02962c8 c384553c 
c02962c8 c0381f10 
[    0.042054] NIP [c00666a0] ftrace_bug+0x13c/0x1c4
[    0.042268] LR [c006782c] ftrace_process_locs+0x1b4/0x2b0
[    0.042405] Call Trace:
[    0.042592] [c0381f10] [c006fce8] ftrace_now+0x30/0x78 (unreliable)
[    0.042889] [c0381f40] [c006782c] ftrace_process_locs+0x1b4/0x2b0
[    0.043182] [c0381f70] [c0339944] ftrace_init+0x1e8/0x21c
[    0.043443] [c0381fb0] [c0333964] start_kernel+0x284/0x2a0
[    0.043704] [c0381ff0] [c0002224] start_here+0x44/0xa8
[    0.043869] Instruction dump:
[    0.044013] 2f9e0003 7f1ed800 3bbd0001 7f44d378 409dffc8 3c60c02f 3863101c 
4bfbbbb9 
[    0.044508] 48000080 3d20c04b 8929bc82 69200001 <0f000000> 2f890000 
40be0010 38000001 
[    0.045206] ---[ end trace 31fd0ba7d8756001 ]---
[    0.045349] ftrace faulted on writing [<c02962c8>] dns_query+0x8/0x278
...


This is on a 2.6.38 kernel (2.6.32 fails too). Debugging shows that 
__copy_tofrom_user() fails with return code 4 (called via 
ftrace_modify_code()).

Before digging deeper: Has anybody tried/tested ftrace on MPC8xx? Any ideas 
what going wrong here?

Thanks.

Best regards,
Stefan

Re: FTrace on MPC8xx

[Joakim Tjernlund]

Stefan Roese <ml@stefan-roese.de> wrote on 2011/04/13 16:32:21:
>
> Hi,
>
> I noticed that ftrace doesn't seem to work on MPC8xx. I'm trying to use it on
> an MPC855T board, but ftrace oopses upon startup:
>
> [    0.028785] ftrace: allocating 10312 entries in 31 pages
> [    0.038594] ------------[ cut here ]------------
> [    0.038760] WARNING: at kernel/trace/ftrace.c:1014
> [    0.038889] Modules linked in:
> [    0.039062] NIP: c00666a0 LR: c006782c CTR: 00000001
> [    0.039249] REGS: c0381e60 TRAP: 0700   Not tainted  (2.6.38-default+)
> [    0.039417] MSR: 00021032 <ME,CE,IR,DR>  CR: 53009393  XER: a000af7f
> [    0.039856] TASK = c0360500[0] 'swapper' THREAD: c0380000
> [    0.040001] GPR00: 00000001 c0381f10 c0360500 ffffffff c02962c8 00000000
> c02962c4 00000000
> [    0.040487] GPR08: 1de1a3e3 00000000 000258e3 c0381f40 93009399 10091a60
> 03fff800 0040055c
> [    0.040977] GPR16: 00000001 00000001 ffffffff 007fff00 03ff9fb0 00000000
> 00000000 c0384080
> [    0.041459] GPR24: c000f234 00000000 024acd00 00009032 c02962c8 c384553c
> c02962c8 c0381f10
> [    0.042054] NIP [c00666a0] ftrace_bug+0x13c/0x1c4
> [    0.042268] LR [c006782c] ftrace_process_locs+0x1b4/0x2b0
> [    0.042405] Call Trace:
> [    0.042592] [c0381f10] [c006fce8] ftrace_now+0x30/0x78 (unreliable)
> [    0.042889] [c0381f40] [c006782c] ftrace_process_locs+0x1b4/0x2b0
> [    0.043182] [c0381f70] [c0339944] ftrace_init+0x1e8/0x21c
> [    0.043443] [c0381fb0] [c0333964] start_kernel+0x284/0x2a0
> [    0.043704] [c0381ff0] [c0002224] start_here+0x44/0xa8
> [    0.043869] Instruction dump:
> [    0.044013] 2f9e0003 7f1ed800 3bbd0001 7f44d378 409dffc8 3c60c02f 3863101c
> 4bfbbbb9
> [    0.044508] 48000080 3d20c04b 8929bc82 69200001 <0f000000> 2f890000
> 40be0010 38000001
> [    0.045206] ---[ end trace 31fd0ba7d8756001 ]---
> [    0.045349] ftrace faulted on writing [<c02962c8>] dns_query+0x8/0x278
> ...
>
>
> This is on a 2.6.38 kernel (2.6.32 fails too). Debugging shows that
> __copy_tofrom_user() fails with return code 4 (called via
> ftrace_modify_code()).
>
> Before digging deeper: Has anybody tried/tested ftrace on MPC8xx? Any ideas
> what going wrong here?

Just an idea, remove the usage of dcbX insn, these has been problematic before.
How big was the size to copy_tofrom_user()? Did it mange to copy any bytes?

 Jocke

Re: FTrace on MPC8xx

[Stefan Roese]

Hi Joakim,

On Wednesday 13 April 2011 16:58:21 Joakim Tjernlund wrote:
> > This is on a 2.6.38 kernel (2.6.32 fails too). Debugging shows that
> > __copy_tofrom_user() fails with return code 4 (called via
> > ftrace_modify_code()).
> > 
> > Before digging deeper: Has anybody tried/tested ftrace on MPC8xx? Any
> > ideas what going wrong here?
> 
> Just an idea, remove the usage of dcbX insn, these has been problematic
> before.

I originally tested with 2.6.32. Same problem there. And your 8xx patches with 
the dcbX changes are not included in 2.6.32 yet.

> How big was the size to copy_tofrom_user()? Did it mange to copy
> any bytes?

The size in __copy_tofrom_user is 4. And its the first call in 
ftrace_modify_code() that fails directly. This works just fine on a PPC440EPx 
board.

Best regards,
Stefan

Re: FTrace on MPC8xx

[Joakim Tjernlund]

Stefan Roese <ml@stefan-roese.de> wrote on 2011/04/13 17:21:33:
>
> Hi Joakim,
>
> On Wednesday 13 April 2011 16:58:21 Joakim Tjernlund wrote:
> > > This is on a 2.6.38 kernel (2.6.32 fails too). Debugging shows that
> > > __copy_tofrom_user() fails with return code 4 (called via
> > > ftrace_modify_code()).
> > >
> > > Before digging deeper: Has anybody tried/tested ftrace on MPC8xx? Any
> > > ideas what going wrong here?
> >
> > Just an idea, remove the usage of dcbX insn, these has been problematic
> > before.
>
> I originally tested with 2.6.32. Same problem there. And your 8xx patches with
> the dcbX changes are not included in 2.6.32 yet.

OK

>
> > How big was the size to copy_tofrom_user()? Did it mange to copy
> > any bytes?
>
> The size in __copy_tofrom_user is 4. And its the first call in
> ftrace_modify_code() that fails directly. This works just fine on a PPC440EPx
> board.

Since the size is only 4 it would not use dcbX anyway(I think).
Then is is probably called with the wrong addresses?

 Jocke

Re: FTrace on MPC8xx

[Stefan Roese]

Hi Joakim,

On Wednesday 13 April 2011 17:38:03 Joakim Tjernlund wrote:
> > > How big was the size to copy_tofrom_user()? Did it mange to copy
> > > any bytes?
> > 
> > The size in __copy_tofrom_user is 4. And its the first call in
> > ftrace_modify_code() that fails directly. This works just fine on a
> > PPC440EPx board.
> 
> Since the size is only 4 it would not use dcbX anyway(I think).
> Then is is probably called with the wrong addresses?

No, addresses seem to be correct. I checked a bit further (I'm quite new to 
the MPC8xx MMU) and it seems that trying to modify the code (that's the 
destination address) via __copy_tofrom_user() fails on MPC8xx. Still not sure 
why this is the case. Perhaps an 8xx guru might chime in here. ;)

BTW: I just noticed that enabling CONFIG_PIN_TLB seems to resolve this issue. 
With this option enabled, the dynamic code modification works just fine.

Joakim, Scott? Any ideas on this?

Thanks,
Stefan

Re: FTrace on MPC8xx

[Joakim Tjernlund]

Stefan Roese <ml@stefan-roese.de> wrote on 2011/04/14 17:59:30:
>
> Hi Joakim,
>
> On Wednesday 13 April 2011 17:38:03 Joakim Tjernlund wrote:
> > > > How big was the size to copy_tofrom_user()? Did it mange to copy
> > > > any bytes?
> > >
> > > The size in __copy_tofrom_user is 4. And its the first call in
> > > ftrace_modify_code() that fails directly. This works just fine on a
> > > PPC440EPx board.
> >
> > Since the size is only 4 it would not use dcbX anyway(I think).
> > Then is is probably called with the wrong addresses?
>
> No, addresses seem to be correct. I checked a bit further (I'm quite new to
> the MPC8xx MMU) and it seems that trying to modify the code (that's the
> destination address) via __copy_tofrom_user() fails on MPC8xx. Still not sure
> why this is the case. Perhaps an 8xx guru might chime in here. ;)
>
> BTW: I just noticed that enabling CONFIG_PIN_TLB seems to resolve this issue.
> With this option enabled, the dynamic code modification works just fine.
>
> Joakim, Scott? Any ideas on this?

hmm, I guess 8xx really maps kernel RO as RO :) Try
changing in pte-8xx.h:
 - #define _PAGE_KERNEL_RO	(_PAGE_SHARED)
 + #define _PAGE_KERNEL_RO	(_PAGE_RW |_PAGE_SHARED)

Re: FTrace on MPC8xx

[Joakim Tjernlund]

>
> Stefan Roese <ml@stefan-roese.de> wrote on 2011/04/14 17:59:30:
> >
> > Hi Joakim,
> >
> > On Wednesday 13 April 2011 17:38:03 Joakim Tjernlund wrote:
> > > > > How big was the size to copy_tofrom_user()? Did it mange to copy
> > > > > any bytes?
> > > >
> > > > The size in __copy_tofrom_user is 4. And its the first call in
> > > > ftrace_modify_code() that fails directly. This works just fine on a
> > > > PPC440EPx board.
> > >
> > > Since the size is only 4 it would not use dcbX anyway(I think).
> > > Then is is probably called with the wrong addresses?
> >
> > No, addresses seem to be correct. I checked a bit further (I'm quite new to
> > the MPC8xx MMU) and it seems that trying to modify the code (that's the
> > destination address) via __copy_tofrom_user() fails on MPC8xx. Still not sure
> > why this is the case. Perhaps an 8xx guru might chime in here. ;)
> >
> > BTW: I just noticed that enabling CONFIG_PIN_TLB seems to resolve this issue.
> > With this option enabled, the dynamic code modification works just fine.
> >
> > Joakim, Scott? Any ideas on this?
>
> hmm, I guess 8xx really maps kernel RO as RO :) Try
> changing in pte-8xx.h:
>  - #define _PAGE_KERNEL_RO   (_PAGE_SHARED)
>  + #define _PAGE_KERNEL_RO   (_PAGE_RW |_PAGE_SHARED)

hmm, I wonder if not this is the problem(in pte-common.h)
#if defined(CONFIG_KGDB) || defined(CONFIG_XMON) || defined(CONFIG_BDI_SWITCH) ||\
	defined(CONFIG_KPROBES)
#define PAGE_KERNEL_TEXT	PAGE_KERNEL_X
#else
#define PAGE_KERNEL_TEXT	PAGE_KERNEL_ROX
#endif

What is PAGE_KERNEL_TEXT for you?
I think it must be PAGE_KERNEL_X, otherwise kernel text will be readonly.

 Jocke

Re: FTrace on MPC8xx

[Stefan Roese]

Hi Joakim,

On Thursday 14 April 2011 21:21:23 Joakim Tjernlund wrote:
> > hmm, I guess 8xx really maps kernel RO as RO :) Try
> > 
> > changing in pte-8xx.h:
> >  - #define _PAGE_KERNEL_RO   (_PAGE_SHARED)
> >  + #define _PAGE_KERNEL_RO   (_PAGE_RW |_PAGE_SHARED)
> 
> hmm, I wonder if not this is the problem(in pte-common.h)
> #if defined(CONFIG_KGDB) || defined(CONFIG_XMON) ||
> defined(CONFIG_BDI_SWITCH) ||\ defined(CONFIG_KPROBES)
> #define PAGE_KERNEL_TEXT	PAGE_KERNEL_X
> #else
> #define PAGE_KERNEL_TEXT	PAGE_KERNEL_ROX
> #endif
> 
> What is PAGE_KERNEL_TEXT for you?
> I think it must be PAGE_KERNEL_X, otherwise kernel text will be readonly.

Yes, that's it! Its PAGE_KERNEL_ROX right now. We need to add CONFIG_FTRACE or 
at least CONFIG_DYNAMIC_FTRACE to the #if statement above.

Do you want to send a patch (since you detected the real problem)? Or should I 
do this?

Thanks.

Best regards,
Stefan

Re: FTrace on MPC8xx

[Joakim Tjernlund]

Stefan Roese <ml@stefan-roese.de> wrote on 2011/04/15 09:22:42:
> Hi Joakim,
>
> On Thursday 14 April 2011 21:21:23 Joakim Tjernlund wrote:
> > > hmm, I guess 8xx really maps kernel RO as RO :) Try
> > >
> > > changing in pte-8xx.h:
> > >  - #define _PAGE_KERNEL_RO   (_PAGE_SHARED)
> > >  + #define _PAGE_KERNEL_RO   (_PAGE_RW |_PAGE_SHARED)
> >
> > hmm, I wonder if not this is the problem(in pte-common.h)
> > #if defined(CONFIG_KGDB) || defined(CONFIG_XMON) ||
> > defined(CONFIG_BDI_SWITCH) ||\ defined(CONFIG_KPROBES)
> > #define PAGE_KERNEL_TEXT   PAGE_KERNEL_X
> > #else
> > #define PAGE_KERNEL_TEXT   PAGE_KERNEL_ROX
> > #endif
> >
> > What is PAGE_KERNEL_TEXT for you?
> > I think it must be PAGE_KERNEL_X, otherwise kernel text will be readonly.
>
> Yes, that's it! Its PAGE_KERNEL_ROX right now. We need to add CONFIG_FTRACE or
> at least CONFIG_DYNAMIC_FTRACE to the #if statement above.
>
> Do you want to send a patch (since you detected the real problem)? Or should I
> do this?

Feel free to send it. I am barely managing my mail ATM.

 Jocke