All of lore.kernel.org
 help / color / mirror / Atom feed
From: Steven Rostedt <rostedt@goodmis.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@us.ibm.com>,
	LKML <linux-kernel@vger.kernel.org>, Tejun Heo <tj@kernel.org>,
	Ingo Molnar <mingo@kernel.org>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	Jiri Olsa <jolsa@redhat.com>
Subject: Re: [RFC][PATCH] ftrace: Use schedule_on_each_cpu() as a heavy synchronize_sched()
Date: Wed, 29 May 2013 09:41:40 -0400	[thread overview]
Message-ID: <1369834900.15552.62.camel@gandalf.local.home> (raw)
In-Reply-To: <20130529075249.GC12193@twins.programming.kicks-ass.net>

On Wed, 2013-05-29 at 09:52 +0200, Peter Zijlstra wrote:

> Just to be clear, its the idle part that's a problem, right?

Actually, it's the userspace part that's triggered the bugs.


>  Being stuck
> in userspace isn't a problem since if that CPU is in userspace its
> certainly not got a reference to whatever list entry we're removing.

The issue is coming out of and going into userspace. For example, we
trace the function user_exit(), which is the function that tells RCU
it's coming back into the kernel. At this point, RCU doesn't keep track
of the preempt disable/enable that's done there.

> 
> Now when the CPU really is idle, its obviously not using tracing either;
> so only the gray area where RCU thinks we're idle but we're not actually
> idle is a problem?

Right. It's going into and coming out of idle or userspace that's the
issue. There's a very small window where RCU is blind to this.

> 
> Is there something a little smarter we can do? Could we use
> on_each_cpu_cond() with a function that checks if the CPU really is
> fully idle?

One thing I thought about doing was both a synchronize_sched() and then
a msleep(5). As a single function trace should never last 5
milliseconds. But to me, that's just hacky, and this is the real
solution.

> 
> > To implement the RCU synchronization, instead of using
> > synchronize_sched() the use of schedule_on_each_cpu() is performed. This
> > means that when a dynamically allocated ftrace_ops, or a control ops is
> > being unregistered, all CPUs must be touched and execute a ftrace_sync()
> > stub function via the work queues. This will rip CPUs out from idle or
> > in dynamic tick mode. This only happens when a user disables perf
> > function tracing or other dynamically allocated function tracers, but it
> > allows us to continue to debug RCU and context tracking with function
> > tracing.
> 
> I don't suppose there's anything perf can do to about this right? Since
> its all on user demand we're kinda stuck with dynamic memory.

Right, and perf isn't the only one. SystemTap, lttng, and even other
parts of ftrace can have this problem. Anyone that does a dynamic
allocation needs a full synchronization.

-- Steve



  parent reply	other threads:[~2013-05-29 13:41 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-05-29  0:01 [RFC][PATCH] ftrace: Use schedule_on_each_cpu() as a heavy synchronize_sched() Steven Rostedt
2013-05-29  7:52 ` Peter Zijlstra
2013-05-29 13:33   ` Paul E. McKenney
2013-05-29 13:55     ` Steven Rostedt
2013-05-29 13:41   ` Steven Rostedt [this message]
2014-06-19  1:56   ` Steven Rostedt
2014-06-19  2:28     ` Paul E. McKenney
2014-06-19  7:18       ` Masami Hiramatsu
2013-05-29  8:23 ` Paul E. McKenney
2013-06-04 11:03 ` Frederic Weisbecker
2013-06-04 12:11   ` Steven Rostedt
2013-06-04 12:30     ` Frederic Weisbecker
2013-06-05 11:51 ` Peter Zijlstra
2013-06-05 13:36   ` Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1369834900.15552.62.camel@gandalf.local.home \
    --to=rostedt@goodmis.org \
    --cc=fweisbec@gmail.com \
    --cc=jolsa@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=paulmck@us.ibm.com \
    --cc=peterz@infradead.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.