All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Jiazi.Li" <jqqlijiazi@gmail.com>
To: Steven Rostedt <rostedt@goodmis.org>, Ingo Molnar <mingo@redhat.com>
Cc: "Jiazi.Li" <jiazi.li@transsion.com>, linux-trace-devel@vger.kernel.org
Subject: [PATCH] ring-buffer: Add barrire in rb_move_tail
Date: Tue, 30 Aug 2022 20:08:54 +0800	[thread overview]
Message-ID: <20220830120854.7545-1-jiazi.li@transsion.com> (raw)

The following crash issue was encountered several times when read
trace_pipe:

1. in ring_buffer_event_data, found event's type is bigger than
RINGBUF_TYPE_DATA_TYPE_LEN_MAX, print WARNING:
[249033.677658]WARNING: CPU: 1 PID: 9975 at ring_buffer_event_data+0x38/0x48

2. trace_read_pipe print this event resulting in crash issue:
[249033.806200] die+0xac/0x37c
[249033.806891] die_kernel_fault+0x60/0x70
[249033.807710] die_kernel_fault+0x0/0x70
[249033.808518] do_bad_area+0x30/0x130
[249033.809296] do_translation_fault+0x34/0x40
[249033.810158] do_mem_abort+0x4c/0xf8
[249033.810935] el1_da+0x1c/0xac
[249033.811647] string+0x38/0x74
[249033.812357] vsnprintf+0x430/0x65c
[249033.813122] seq_buf_vprintf+0x64/0xcc
[249033.813934] trace_seq_printf+0x70/0xd8
[249033.814755] trace_raw_output_ipi_handler+0x40/0x54
[249033.815705] print_trace_line+0x3ec/0x424
[249033.816547] tracing_read_pipe+0x28c/0x3ec
[249033.817402] __vfs_read+0x44/0x13c
[249033.818166] vfs_read+0xb8/0x150
[249033.818908] ksys_read+0x6c/0xcc
[249033.819651] __arm64_sys_read+0x18/0x20
[249033.820470] el0_svc_common+0x98/0x160
[249033.821278] el0_svc_handler+0x60/0x78
[249033.822086] el0_svc+0x8/0x300

ipi event is not enable in my device, the event tracing_read_pipe
want to print is a PADDING event generated in rb_move_tail.
array[0] of this event is 0x18, ipi entry trace event's type value
is 0x18 too in my device.
so trace_read_pipe call trace_raw_output_ipi_handler to print this
event, eventually, system crash because illegal pointers.

In ring_buffer_peek, PADDING type event will not return to caller.
So at least when ring_buffer_peek is called, this event has not
been set to PADDING.

My reader task poll on trace_pipe and other files, wakeup
by other files and found that trace ring_buffer is not empty
because of read != commit.
But maybe because of cpu out-of-order execution, new event has not
been set to PADDING type, and leads to above issue.

Can we fix this issue by add a barrier between rb_reset_tail
and rb_end_commit?
Or are there other bugs that cause this issue?

Signed-off-by: Jiazi.Li <jiazi.li@transsion.com>
---
 kernel/trace/ring_buffer.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index d59b6a328b7f..20f3ef839467 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -2717,6 +2717,11 @@ rb_move_tail(struct ring_buffer_per_cpu *cpu_buffer,
 
 	rb_reset_tail(cpu_buffer, tail, info);
 
+	/*
+	 * Add barrier to make sure tail event be set to
+	 * PADDING type before update commit.
+	 */
+	smp_mb();
 	/* Commit what we have for now. */
 	rb_end_commit(cpu_buffer);
 	/* rb_end_commit() decs committing */
-- 
2.17.1


             reply	other threads:[~2022-08-30 12:09 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-08-30 12:08 Jiazi.Li [this message]
2022-09-01 15:13 ` [PATCH] ring-buffer: Add barrire in rb_move_tail Steven Rostedt
2022-09-02  3:47   ` lijiazi
2022-09-02 12:56     ` Steven Rostedt
2022-09-05  3:23       ` lijiazi
2022-09-06 16:25         ` Steven Rostedt
2022-09-07  3:03           ` lijiazi
2022-09-07  8:13           ` lijiazi
2022-09-15  8:25             ` lijiazi
2022-09-28 14:54               ` Steven Rostedt
2022-09-28 14:59                 ` Steven Rostedt
2022-09-28 20:29                   ` Steven Rostedt
2022-09-29  9:12                     ` lijiazi
2022-09-30 21:49                       ` Steven Rostedt
2022-10-08  1:21                         ` lijiazi
2022-10-08 12:50                           ` Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220830120854.7545-1-jiazi.li@transsion.com \
    --to=jqqlijiazi@gmail.com \
    --cc=jiazi.li@transsion.com \
    --cc=linux-trace-devel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=rostedt@goodmis.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.