archive mirror
 help / color / mirror / Atom feed
From: Steven Rostedt <>
Cc: Ingo Molnar <>,
	Andrew Morton <>,, Stefan Metzmacher <>
Subject: [for-linus][PATCH 7/7] tracepoints: Update static_call before tp_funcs when adding a tracepoint
Date: Fri, 23 Jul 2021 08:55:01 -0400	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

From: "Steven Rostedt (VMware)" <>

Because of the significant overhead that retpolines pose on indirect
calls, the tracepoint code was updated to use the new "static_calls" that
can modify the running code to directly call a function instead of using
an indirect caller, and this function can be changed at runtime.

In the tracepoint code that calls all the registered callbacks that are
attached to a tracepoint, the following is done:

	it_func_ptr = rcu_dereference_raw((&__tracepoint_##name)->funcs);
	if (it_func_ptr) {
		__data = (it_func_ptr)->data;
		static_call(tp_func_##name)(__data, args);

If there's just a single callback, the static_call is updated to just call
that callback directly. Once another handler is added, then the static
caller is updated to call the iterator, that simply loops over all the
funcs in the array and calls each of the callbacks like the old method
using indirect calling.

The issue was discovered with a race between updating the funcs array and
updating the static_call. The funcs array was updated first and then the
static_call was updated. This is not an issue as long as the first element
in the old array is the same as the first element in the new array. But
that assumption is incorrect, because callbacks also have a priority
field, and if there's a callback added that has a higher priority than the
callback on the old array, then it will become the first callback in the
new array. This means that it is possible to call the old callback with
the new callback data element, which can cause a kernel panic.

	static_call = callback1()
	funcs[] = {callback1,data1};
	callback2 has higher priority than callback1

	CPU 1				CPU 2
	-----				-----

   new_funcs = {callback2,data2},

   rcu_assign_pointer(tp->funcs, new_funcs);

   * Now tp->funcs has the new array
   * but the static_call still calls callback1

				it_func_ptr = tp->funcs [ new_funcs ]
				data = it_func_ptr->data [ data2 ]
				static_call(callback1, data);

				/* Now callback1 is called with
				 * callback2's data */



To prevent this from happening, always switch the static_call to the
iterator before assigning the tp->funcs to the new array. The iterator will
always properly match the callback with its data.

To trigger this bug:

  In one terminal:

    while :; do hackbench 50; done

  In another terminal

    echo 1 > /sys/kernel/tracing/events/sched/sched_waking/enable
    while :; do
        echo 1 > /sys/kernel/tracing/set_event_pid;
        sleep 0.5
        echo 0 > /sys/kernel/tracing/set_event_pid;
        sleep 0.5

And it doesn't take long to crash. This is because the set_event_pid adds
a callback to the sched_waking tracepoint with a high priority, which will
be called before the sched_waking trace event callback is called.

Note, the removal to a single callback updates the array first, before
changing the static_call to single callback, which is the proper order as
the first element in the array is the same as what the static_call is
being changed to.


Fixes: d25e37d89dd2f ("tracepoint: Optimize using static_call()")
Reported-by: Stefan Metzmacher <>
tested-by: Stefan Metzmacher <>
Signed-off-by: Steven Rostedt (VMware) <>
 kernel/tracepoint.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c
index 976bf8ce8039..fc32821f8240 100644
--- a/kernel/tracepoint.c
+++ b/kernel/tracepoint.c
@@ -299,8 +299,8 @@ static int tracepoint_add_func(struct tracepoint *tp,
 	 * a pointer to it.  This array is referenced by __DO_TRACE from
 	 * include/linux/tracepoint.h using rcu_dereference_sched().
-	rcu_assign_pointer(tp->funcs, tp_funcs);
 	tracepoint_update_call(tp, tp_funcs, false);
+	rcu_assign_pointer(tp->funcs, tp_funcs);

      parent reply	other threads:[~2021-07-23 12:56 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-07-23 12:54 [for-linus][PATCH 0/7] tracing: Fixes for 5.14-rc2 Steven Rostedt
2021-07-23 12:54 ` [for-linus][PATCH 1/7] tracing: Fix bug in rb_per_cpu_empty() that might cause deadloop Steven Rostedt
2021-07-23 12:54 ` [for-linus][PATCH 2/7] tracing: Synthetic event field_pos is an index not a boolean Steven Rostedt
2021-07-23 12:54 ` [for-linus][PATCH 3/7] tracing/histogram: Rename "cpu" to "common_cpu" Steven Rostedt
2021-07-23 12:54 ` [for-linus][PATCH 4/7] tracing: Clean up alloc_synth_event() Steven Rostedt
2021-07-23 12:54 ` [for-linus][PATCH 5/7] ftrace: Avoid synchronize_rcu_tasks_rude() call when not necessary Steven Rostedt
2021-07-23 12:55 ` [for-linus][PATCH 6/7] ftrace: Remove redundant initialization of variable ret Steven Rostedt
2021-07-23 12:55 ` Steven Rostedt [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \
    --subject='Re: [for-linus][PATCH 7/7] tracepoints: Update static_call before tp_funcs when adding a tracepoint' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).