Exception in function_graph tracing

Exception in function_graph tracing

[Danilo Cezar Zanella]

Hi

I'm getting the following error message when I set the current_tracer
to function_graph. Procedures:

cd / sys/kernel/tracing
echo function_graph > current_tracer

Then I saw the following message in the console.

Unable to handle kernel paging request at virtual address fffffffc
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at kernel/sched/core.c:7270 migrate_disable+0x104/0x114
Modules linked in: rtc_armada38x ts7800_v2_wdt spi_orion spi_gpio
i2c_gpio i2c_algo_bit
CPU: 0 PID: 0 Comm:  Not tainted 4.19.249-rt111-llama-rt+ #36
Hardware name: Marvell Armada 380/385 (Device Tree)
[<c011424c>] (unwind_backtrace) from [<c010e318>] (show_stack+0x18/0x1c)
[<c010e318>] (show_stack) from [<c09da8ac>] (dump_stack+0x84/0x98)
[<c09da8ac>] (dump_stack) from [<c0124bc8>] (__warn+0x100/0x118)
[<c0124bc8>] (__warn) from [<c0124d0c>] (warn_slowpath_null+0x48/0x50)
[<c0124d0c>] (warn_slowpath_null) from [<c015163c>]
(migrate_disable+0x104/0x114)
[<c015163c>] (migrate_disable) from [<c0182920>] (vprintk_emit+0x10c/0x1ec)
[<c0182920>] (vprintk_emit) from [<c0182bc0>] (vprintk_default+0x28/0x30)
[<c0182bc0>] (vprintk_default) from [<c09d5100>] (printk+0x38/0x5c)
[<c09d5100>] (printk) from [<c09d4dc8>] (__do_kernel_fault.part.0+0x4c/0x7c)
[<c09d4dc8>] (__do_kernel_fault.part.0) from [<c0118d84>]
(do_page_fault+0x380/0x394)
[<c0118d84>] (do_page_fault) from [<c0118f6c>] (do_DataAbort+0x50/0xf0)
---[ end trace 0000000000000002 ]---
pgd = 721da14e
[fffffffc] *pgd=2fffd861, *pte=00000000, *ppte=00000000
Internal error: Oops: 37 [#1] PREEMPT SMP ARM
Modules linked in: rtc_armada38x ts7800_v2_wdt spi_orion spi_gpio
i2c_gpio i2c_algo_bit
CPU: 0 PID: 0 Comm:  Tainted: G        W         4.19.249-rt111-llama-rt+ #36
Hardware name: Marvell Armada 380/385 (Device Tree)
PC is at prepare_ftrace_return+0x20/0x48
LR is at ftrace_graph_caller+0x18/0x20
pc : [<c0112884>]    lr : [<c0112540>]    psr: 600f0193
sp : ef12a090  ip : 00000051  fp : 00000000
r10: e9c61d7c  r9 : ef12a000  r8 : 00000001
r7 : ef12a19c  r6 : ffffffff  r5 : 600f0193  r4 : c0112884
r3 : 00000000  r2 : 00000000  r1 : c0118f28  r0 : fffffffc
Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 10c5387d  Table: 29c7004a  DAC: 00000051
Process  (pid: 0, stack limit = 0x6b08bd12)
Stack: (0xef12a090 to 0xef07651c)
[<c0112884>] (prepare_ftrace_return) from [<c0112540>]
(ftrace_graph_caller+0x18/0x20)
Code: e5933010 e593366c e3530000 18bd8070 (e5905000)
---[ end trace 0000000000000003 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
SMP: failed to stop secondary CPUs

What is going on?
Am I doing something wrong?
Thanks

Danilo

Re: Exception in function_graph tracing

[Sebastian Andrzej Siewior]

On 2022-09-13 16:21:27 [-0300], Danilo Cezar Zanella wrote:
> Hi
Hi,

> I'm getting the following error message when I set the current_tracer
> to function_graph. Procedures:
> 
> cd / sys/kernel/tracing
> echo function_graph > current_tracer
> 
> Then I saw the following message in the console.
> 
> Unable to handle kernel paging request at virtual address fffffffc
…
> What is going on?
> Am I doing something wrong?

No, not as far I can tell. It looks like the function_graph tracer is
busted in v4.19 even without the PREEMPT_RT patch. The function tracer
seems to work.

> Thanks
> 
> Danilo

Sebastian

Re: Exception in function_graph tracing

[Sebastian Andrzej Siewior]

On 2022-09-16 17:28:00 [-0300], Danilo Cezar Zanella wrote:
> Hi Sebastian,
Hi Danilo,

please don't lose the list.

> > No, not as far I can tell.
> It sounds good. ;-)
> 
> > It looks like the function_graph tracer is busted in v4.19 even without the PREEMPT_RT patch.
> It sounds bad... :-(
> 
> I'm new in LKM development and even more in Kernel tracing.
> I've tested 'nop' and 'function' tracing, both seem to work. Do you
> know when this issue was solved? Perhaps could I do a backport of
> it(if it is easy, of course)?

I managed to reproduce it on v4.19 but on v6.0-rcX. So it must have
been fixed somehow in between. I browsed the history but I did not
notice anything obvious.
So someone needs to dig deeper…

> Thank you again
> 
> Danilo

Sebastian

[PATCH] ARM: fix function graph tracer and unwinder dependencies

[Sebastian Andrzej Siewior]

Danilo Cezar Zanella reported broken function graph tracer in the v4.19
tree. This was caused by the backport of commit
   f9b58e8c7d031 ("ARM: 8800/1: use choice for kernel unwinders")

which ended in the v4.19-stable tree as of v4.19.222.
It is also part of v4.14-stable since v4.14.259.
It is also part of v4.9-stable since v4.9.299.

------->8---------

From: Russell King <rmk+kernel@armlinux.org.uk>
Date: Tue, 23 Apr 2019 17:09:38 +0100
Subject: [PATCH] ARM: fix function graph tracer and unwinder dependencies

Upstream commit 503621628b32782a07b2318e4112bd4372aa3401

Naresh Kamboju recently reported that the function-graph tracer crashes
on ARM. The function-graph tracer assumes that the kernel is built with
frame pointers.

We explicitly disabled the function-graph tracer when building Thumb2,
since the Thumb2 ABI doesn't have frame pointers.

We recently changed the way the unwinder method was selected, which
seems to have made it more likely that we can end up with the function-
graph tracer enabled but without the kernel built with frame pointers.

Fix up the function graph tracer dependencies so the option is not
available when we have no possibility of having frame pointers, and
adjust the dependencies on the unwinder option to hide the non-frame
pointer unwinder options if the function-graph tracer is enabled.

Reviewed-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Masami Hiramatsu <mhiramat@kernel.org>
Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reported-by: Danilo Cezar Zanella <danilo.zanella@iag.usp.br>
---
 arch/arm/Kconfig       | 2 +-
 arch/arm/Kconfig.debug | 6 +++---
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index d89d013f586cb..fce7e85f3ef57 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -68,7 +68,7 @@ config ARM
 	select HAVE_EFFICIENT_UNALIGNED_ACCESS if (CPU_V6 || CPU_V6K || CPU_V7) && MMU
 	select HAVE_EXIT_THREAD
 	select HAVE_FTRACE_MCOUNT_RECORD if (!XIP_KERNEL)
-	select HAVE_FUNCTION_GRAPH_TRACER if (!THUMB2_KERNEL)
+	select HAVE_FUNCTION_GRAPH_TRACER if (!THUMB2_KERNEL && !CC_IS_CLANG)
 	select HAVE_FUNCTION_TRACER if (!XIP_KERNEL)
 	select HAVE_FUTEX_CMPXCHG if FUTEX
 	select HAVE_GCC_PLUGINS
diff --git a/arch/arm/Kconfig.debug b/arch/arm/Kconfig.debug
index 01c760929c9e4..b931fac129a1b 100644
--- a/arch/arm/Kconfig.debug
+++ b/arch/arm/Kconfig.debug
@@ -47,8 +47,8 @@ config DEBUG_WX
 
 choice
 	prompt "Choose kernel unwinder"
-	default UNWINDER_ARM if AEABI && !FUNCTION_GRAPH_TRACER
-	default UNWINDER_FRAME_POINTER if !AEABI || FUNCTION_GRAPH_TRACER
+	default UNWINDER_ARM if AEABI
+	default UNWINDER_FRAME_POINTER if !AEABI
 	help
 	  This determines which method will be used for unwinding kernel stack
 	  traces for panics, oopses, bugs, warnings, perf, /proc/<pid>/stack,
@@ -65,7 +65,7 @@ config UNWINDER_FRAME_POINTER
 
 config UNWINDER_ARM
 	bool "ARM EABI stack unwinder"
-	depends on AEABI
+	depends on AEABI && !FUNCTION_GRAPH_TRACER
 	select ARM_UNWIND
 	help
 	  This option enables stack unwinding support in the kernel
-- 
2.37.2

Re: [PATCH] ARM: fix function graph tracer and unwinder dependencies

[Greg KH]

On Wed, Oct 05, 2022 at 10:43:34AM +0200, Sebastian Andrzej Siewior wrote:
> Danilo Cezar Zanella reported broken function graph tracer in the v4.19
> tree. This was caused by the backport of commit
>    f9b58e8c7d031 ("ARM: 8800/1: use choice for kernel unwinders")
> 
> which ended in the v4.19-stable tree as of v4.19.222.
> It is also part of v4.14-stable since v4.14.259.
> It is also part of v4.9-stable since v4.9.299.
> 

Now queued up, thanks.

greg k-h