Can I use ftrace to analysis cobalt kernel

Can I use ftrace to analysis cobalt kernel

[Ivan Jiang]

Dear guys:

 

       Can I use ftrace to analysis the kernel with patch of ipipe.

       I find if use function_graph whe kernel will crash like below:

       echo function_graph > current_tracer

root@smarc-rzg2l:/sys/kernel/debug/tracing# [  377.392724] Kernel panic - not syncing: kernel stack overflow

[  377.399125] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.19.140-cip33-g6634f0243a91 #3

[  377.407929] Hardware name: RZG2L Evaluation Board Kit (Discrete Power System Ver.) (Dual Cortex-A55) (DT)

[  377.418424] necro domain: Linux

[  377.422006] Call trace:

[  377.424856]  dump_backtrace+0x0/0x168

[  377.429003]  show_stack+0x24/0x30

[  377.432790]  dump_stack+0xd0/0x110

[  377.436660]  panic+0x138/0x2a8

[  377.440157]  nmi_panic+0x7c/0x80

[  377.443840]  handle_bad_stack+0x120/0x138

[  377.448356]  __bad_stack+0x88/0x8c

[  377.452224]  sched_clock+0x0/0x90

[  377.455998]  function_graph_enter+0x74/0x138

[  377.460797]  prepare_ftrace_return+0x28/0x50

[  377.465593]  ftrace_graph_caller+0x1c/0x24

[  377.470206]  ipipe_test_root+0x10/0x38

[  377.474447]  preempt_schedule_notrace+0x28/0x150

[  377.479624]  trace_clock_local+0x44/0x58

[  377.484046]  function_graph_enter+0x74/0x138

[  377.488839]  prepare_ftrace_return+0x28/0x50

[  377.493637]  ftrace_graph_caller+0x1c/0x24

[  379.179942]  ipipe_test_root+0x10/0x38

[  379.184182]  preempt_schedule_notrace+0x28/0x150

[  379.189343]  trace_clock_local+0x44/0x58

[  379.193766]  function_graph_enter+0x74/0x138

[  379.198563]  prepare_ftrace_return+0x28/0x50

[  379.203358]  ftrace_graph_caller+0x1c/0x24

[  379.207968]  ipipe_test_root+0x10/0x38

[  379.212207]  preempt_schedule_notrace+0x28/0x150

[  379.217378]  trace_clock_local+0x44/0x58

[  379.221799]  function_graph_enter+0x74/0x138

[  379.226593]  prepare_ftrace_return+0x28/0x50

[  379.231391]  ftrace_graph_caller+0x1c/0x24

[  379.236004]  ipipe_test_root+0x10/0x38

[  379.240244]  preempt_schedule_notrace+0x28/0x150

[  379.245415]  ftrace_ops_no_ops+0x1ac/0x1b0

[  379.250025]  skip_ftrace_call+0x0/0x30

[  379.254260]  ipipe_test_root+0x10/0x38

[  379.258501]  preempt_schedule_notrace+0x28/0x150

[  379.263670]  trace_clock_local+0x44/0x58

[  379.268093]  ftrace_return_to_handler+0x78/0x128

[  379.273261]  return_to_handler+0x1c/0x38

[  379.277677]  __do_softirq+0x23c/0x3d0

[  379.281822]  run_ksoftirqd+0x40/0x50

[  379.285874]  kthread+0x130/0x138

[  379.289551]  ret_from_fork+0x14/0x20

[  379.293651] SMP: stopping secondary CPUs

[  379.298123] Kernel Offset: disabled

[  379.302072] CPU features: 0x4,22002238

[  379.306299] Memory Limit: none

[  379.310010] ---[ end Kernel panic - not syncing: kernel stack overflow ]---

[  379.317786] Insufficient stack space to handle exception!

[  379.317830] ESR: 0x96000047 -- DABT (current EL)

[  379.328901] FAR: 0xffff00000805ffc0

[  379.332867] Task stack:     [0xffff000008060000..0xffff000008064000]

[  379.339903] IRQ stack:      [0xffff000008000000..0xffff000008004000]

[  379.346945] Overflow stack: [0xffff80007ff95290..0xffff80007ff96290]

[  379.353984] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.19.140-cip33-g6634f0243a91 #3

[  379.362818] Hardware name: RZG2L Evaluation Board Kit (Discrete Power System Ver.) (Dual Cortex-A55) (DT)

[  379.373333] necro domain: Linux

[  379.376921] pstate: 00c00005 (nzcv daif +PAN +UAO)

[  379.382282] pc : sched_clock+0x0/0x90

[  379.386428] lr : trace_clock_local+0x20/0x58

[  379.391227] sp : ffff000008060000

[  379.394995] x29: ffff000008060000 x28: ffff0000095b60c0 

[  379.400978] x27: ffff00000959c000 x26: 0000000000000100 

[  379.406958] x25: 0000000000000040 x24: ffff0000095b6110 

[  379.412940] x23: ffff0000095b9b38 x22: ffff000008df9b70 

[  379.418923] x21: ffff0000081c03d4 x20: ffff0000095b9748 

[  379.424904] x19: ffff80007dbe6e00 x18: 0000000000000000 

[  379.430885] x17: 0000000000000000 x16: 0000000000000000 

[  379.436869] x15: 0000000000000000 x14: 0000000000000000 

[  379.442852] x13: 0000000000000000 x12: 0000000000000000 

[  379.448839] x11: 0000000000000000 x10: 0000000000000f90 

[  379.454822] x9 : ffff80007d806400 x8 : 0000000000000018 

[  379.460805] x7 : ffff8000776d0fa0 x6 : 0000000015d5de14 

[  379.466793] x5 : 00ffffffffffffff x4 : 0000000000000000 

[  379.472781] x3 : 0000000000000000 x2 : 0000000000000041 

[  379.478757] x1 : ffff0000081c03d4 x0 : 0000000000000001

 

       I was wondering if there’s some option I should use different from normal linux.

       Thank you.

 

       BR.,

       Ivan

Re: Can I use ftrace to analysis cobalt kernel

[Jan Kiszka]

On 24.08.21 14:48, Ivan Jiang via Xenomai wrote:
> Dear guys:
> 
>  
> 
>        Can I use ftrace to analysis the kernel with patch of ipipe.
> 
>        I find if use function_graph whe kernel will crash like below:
> 
>        echo function_graph > current_tracer
> 
> root@smarc-rzg2l:/sys/kernel/debug/tracing# [  377.392724] Kernel panic - not syncing: kernel stack overflow
> 
> [  377.399125] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.19.140-cip33-g6634f0243a91 #3
> 
> [  377.407929] Hardware name: RZG2L Evaluation Board Kit (Discrete Power System Ver.) (Dual Cortex-A55) (DT)
> 

That's ARM64 and I-pipe - might not have been tested / fully enabled in
that combination. Try 5.10 with the dovetail patch, there ftrace is
known to work better. Alternatively: the I-pipe tracer should work on
arm64 (though without a nice function graph).

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: Can I use ftrace to analysis cobalt kernel

[Greg Gallagher]

On Tue, Aug 24, 2021 at 9:22 AM Jan Kiszka via Xenomai <xenomai@xenomai.org>
wrote:

> On 24.08.21 14:48, Ivan Jiang via Xenomai wrote:
> > Dear guys:
> >
> >
> >
> >        Can I use ftrace to analysis the kernel with patch of ipipe.
> >
> >        I find if use function_graph whe kernel will crash like below:
> >
> >        echo function_graph > current_tracer
> >
> > root@smarc-rzg2l:/sys/kernel/debug/tracing# [  377.392724] Kernel panic
> - not syncing: kernel stack overflow
> >
> > [  377.399125] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted
> 4.19.140-cip33-g6634f0243a91 #3
> >
> > [  377.407929] Hardware name: RZG2L Evaluation Board Kit (Discrete Power
> System Ver.) (Dual Cortex-A55) (DT)
> >
>
> That's ARM64 and I-pipe - might not have been tested / fully enabled in
> that combination. Try 5.10 with the dovetail patch, there ftrace is
> known to work better. Alternatively: the I-pipe tracer should work on
> arm64 (though without a nice function graph).
>
> Jan
>
> --
> Siemens AG, T RDA IOT
> Corporate Competence Center Embedded Linux


When you enable ftrace, do you also enable the larger stack size in the
ftrace config?

-Greg

>
>
>

Re: Can I use ftrace to analysis cobalt kernel

[Ivan Jiang]

Hi，Gerg：

 

       Already set the stake size to 1408, but still crash after few minutes.

 

Thanks.

       

 

 

 

On Tue, Aug 24, 2021 at 9:22 AM Jan Kiszka via Xenomai <xenomai@xenomai.org> wrote:

On 24.08.21 14:48, Ivan Jiang via Xenomai wrote:
> Dear guys:
> 
>  
> 
>        Can I use ftrace to analysis the kernel with patch of ipipe.
> 
>        I find if use function_graph whe kernel will crash like below:
> 
>        echo function_graph > current_tracer
> 
> root@smarc-rzg2l:/sys/kernel/debug/tracing# [  377.392724] Kernel panic - not syncing: kernel stack overflow
> 
> [  377.399125] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.19.140-cip33-g6634f0243a91 #3
> 
> [  377.407929] Hardware name: RZG2L Evaluation Board Kit (Discrete Power System Ver.) (Dual Cortex-A55) (DT)
> 

That's ARM64 and I-pipe - might not have been tested / fully enabled in
that combination. Try 5.10 with the dovetail patch, there ftrace is
known to work better. Alternatively: the I-pipe tracer should work on
arm64 (though without a nice function graph).

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

 

When you enable ftrace, do you also enable the larger stack size in the ftrace config?

 

-Greg