All of lore.kernel.org
 help / color / mirror / Atom feed
From: Peter Zijlstra <peterz@infradead.org>
To: 王贇 <yun.wang@linux.alibaba.com>
Cc: Ingo Molnar <mingo@redhat.com>,
	Arnaldo Carvalho de Melo <acme@kernel.org>,
	Mark Rutland <mark.rutland@arm.com>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Jiri Olsa <jolsa@redhat.com>, Namhyung Kim <namhyung@kernel.org>,
	Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Andrii Nakryiko <andrii@kernel.org>,
	Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
	Yonghong Song <yhs@fb.com>,
	John Fastabend <john.fastabend@gmail.com>,
	KP Singh <kpsingh@kernel.org>,
	"open list:PERFORMANCE EVENTS SUBSYSTEM" 
	<linux-perf-users@vger.kernel.org>,
	"open list:PERFORMANCE EVENTS SUBSYSTEM" 
	<linux-kernel@vger.kernel.org>,
	"open list:BPF (Safe dynamic programs and tools)" 
	<netdev@vger.kernel.org>,
	"open list:BPF (Safe dynamic programs and tools)" 
	<bpf@vger.kernel.org>
Subject: Re: [RFC PATCH] perf: fix panic by mark recursion inside perf_log_throttle
Date: Mon, 13 Sep 2021 12:24:24 +0200	[thread overview]
Message-ID: <YT8m2B6D2yWc5Umq@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <f38987a5-dc36-a20d-8c5e-81e8ead5b4dc@linux.alibaba.com>

On Mon, Sep 13, 2021 at 11:00:47AM +0800, 王贇 wrote:
> 
> 
> On 2021/9/10 下午11:38, Peter Zijlstra wrote:
> > On Thu, Sep 09, 2021 at 11:13:21AM +0800, 王贇 wrote:
> >> When running with ftrace function enabled, we observed panic
> >> as below:
> >>
> >>   traps: PANIC: double fault, error_code: 0x0
> >>   [snip]
> >>   RIP: 0010:perf_swevent_get_recursion_context+0x0/0x70
> >>   [snip]
> >>   Call Trace:
> >>    <NMI>
> >>    perf_trace_buf_alloc+0x26/0xd0
> >>    perf_ftrace_function_call+0x18f/0x2e0
> >>    kernelmode_fixup_or_oops+0x5/0x120
> >>    __bad_area_nosemaphore+0x1b8/0x280
> >>    do_user_addr_fault+0x410/0x920
> >>    exc_page_fault+0x92/0x300
> >>    asm_exc_page_fault+0x1e/0x30
> >>   RIP: 0010:__get_user_nocheck_8+0x6/0x13
> >>    perf_callchain_user+0x266/0x2f0
> >>    get_perf_callchain+0x194/0x210
> >>    perf_callchain+0xa3/0xc0
> >>    perf_prepare_sample+0xa5/0xa60
> >>    perf_event_output_forward+0x7b/0x1b0
> >>    __perf_event_overflow+0x67/0x120
> >>    perf_swevent_overflow+0xcb/0x110
> >>    perf_swevent_event+0xb0/0xf0
> >>    perf_tp_event+0x292/0x410
> >>    perf_trace_run_bpf_submit+0x87/0xc0
> >>    perf_trace_lock_acquire+0x12b/0x170
> >>    lock_acquire+0x1bf/0x2e0
> >>    perf_output_begin+0x70/0x4b0
> >>    perf_log_throttle+0xe2/0x1a0
> >>    perf_event_nmi_handler+0x30/0x50
> >>    nmi_handle+0xba/0x2a0
> >>    default_do_nmi+0x45/0xf0
> >>    exc_nmi+0x155/0x170
> >>    end_repeat_nmi+0x16/0x55
> > 
> > kernel/events/Makefile has:
> > 
> > ifdef CONFIG_FUNCTION_TRACER
> > CFLAGS_REMOVE_core.o = $(CC_FLAGS_FTRACE)
> > endif
> > 
> > Which, afaict, should avoid the above, no?
> 
> I'm afraid it's not working for this case, the
> start point of tracing is at lock_acquire() which
> is not from 'kernel/events/core', the following PF
> related function are also not from 'core', prevent
> ftrace on 'core' can't prevent this from happen...

I'm confused tho; where does the #DF come from? Because taking a #PF
from NMI should be perfectly fine.

AFAICT that callchain is something like:

	NMI
	  perf_event_nmi_handler()
	    (part of the chain is missing here)
	      perf_log_throttle()
	        perf_output_begin() /* events/ring_buffer.c */
		  rcu_read_lock()
		    rcu_lock_acquire()
		      lock_acquire()
		        trace_lock_acquire() --> perf_trace_foo

			  ...
			    perf_callchain()
			      perf_callchain_user()
			        #PF (fully expected during a userspace callchain)
				  (some stuff, until the first __fentry)
				    perf_trace_function_call
				      perf_trace_buf_alloc()
				        perf_swevent_get_recursion_context()
					  *BOOM*

Now, supposedly we then take another #PF from get_recursion_context() or
something, but that doesn't make sense. That should just work...

Can you figure out what's going wrong there? going with the RIP, this
almost looks like 'swhash->recursion' goes splat, but again that makes
no sense, that's a per-cpu variable.



  parent reply	other threads:[~2021-09-13 10:24 UTC|newest]

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-09  3:13 [RFC PATCH] perf: fix panic by mark recursion inside perf_log_throttle 王贇
2021-09-09  6:10 ` 王贇
2021-09-10 15:38 ` Peter Zijlstra
2021-09-13  3:00   ` 王贇
2021-09-13  3:21     ` 王贇
2021-09-13 10:24     ` Peter Zijlstra [this message]
2021-09-13 10:36       ` Peter Zijlstra
2021-09-14  2:02         ` 王贇
2021-09-14  1:58       ` 王贇
2021-09-14 10:28         ` Peter Zijlstra
2021-09-15  1:51           ` 王贇
2021-09-15 15:17             ` [PATCH] x86/dumpstack/64: Add guard pages to stack_info Peter Zijlstra
2021-09-16  3:34               ` 王贇
2021-09-16  3:47               ` 王贇
2021-09-16  8:00                 ` Peter Zijlstra
2021-09-16  8:03                   ` Peter Zijlstra
2021-09-16 10:02                     ` Peter Zijlstra
2021-09-17  2:15                       ` 王贇
2021-09-17  3:02                       ` 王贇
2021-09-17 10:21                         ` Peter Zijlstra
2021-09-17 16:40                           ` Peter Zijlstra
2021-09-18  2:30                             ` 王贇
2021-09-18  6:56                               ` Peter Zijlstra
2021-09-18  2:38                             ` 王贇
2021-09-13  3:30 ` [PATCH] perf: fix panic by disable ftrace on fault.c 王贇
2021-09-13 14:49   ` Dave Hansen
2021-09-14  1:52     ` 王贇
2021-09-14  3:02       ` 王贇
2021-09-14  7:23         ` 王贇
2021-09-14 16:16           ` Dave Hansen
2021-09-15  1:56             ` 王贇
2021-09-15  3:27               ` Dave Hansen
2021-09-15  7:22                 ` 王贇
2021-09-15  7:34                   ` 王贇
2021-09-15 15:19                     ` [PATCH] x86: Increase exception stack sizes Peter Zijlstra
2021-09-16  3:42                       ` 王贇
2021-09-21  7:28                       ` [tip: x86/core] " tip-bot2 for Peter Zijlstra
2021-09-21 12:41                       ` tip-bot2 for Peter Zijlstra
2021-09-14  2:08     ` [PATCH] perf: fix panic by disable ftrace on fault.c 王贇

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YT8m2B6D2yWc5Umq@hirez.programming.kicks-ass.net \
    --to=peterz@infradead.org \
    --cc=acme@kernel.org \
    --cc=alexander.shishkin@linux.intel.com \
    --cc=andrii@kernel.org \
    --cc=ast@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=john.fastabend@gmail.com \
    --cc=jolsa@redhat.com \
    --cc=kafai@fb.com \
    --cc=kpsingh@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-perf-users@vger.kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=mingo@redhat.com \
    --cc=namhyung@kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=songliubraving@fb.com \
    --cc=yhs@fb.com \
    --cc=yun.wang@linux.alibaba.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.