Re: [RFC][PATCH] ftrace: Use schedule_on_each_cpu() as a heavy synchronize_sched()

[Steven Rostedt <rostedt@goodmis.org>]

On Wed, 2013-05-29 at 06:33 -0700, Paul E. McKenney wrote:

> > Is there something a little smarter we can do? Could we use
> > on_each_cpu_cond() with a function that checks if the CPU really is
> > fully idle?
> 
> One recent change that should help is making the _rcuidle variants of
> the tracing functions callable from both idle and irq.  To make the
> on_each_cpu_cond() approach work, event tracing would need to switch
> from RCU (which might be preemptible RCU) to RCU-sched (whose read-side
> critical sections can pair with on_each_cpu().  I have to defer to Steven
> on whether this is a good approach.
> 

Just to be clear, the issue is only with the function tracer. This has
nothing to do with trace events, as we have the _rcuidle() variants to
deal with that.

We want the function tracer to trace pretty much everything it can,
especially a complex system like RCU. Thus, I would say that the burden
goes onto the tracing facility to solve this and not prevent tracing
critical parts of RCU.

As you stated with the problem of in_irq(), there's a point where we are
in an interrupt but the in_irq() isn't set yet. And this even shows up
in function tracing:

  <idle>-0       0d...   141.555326: function:             smp_apic_timer_interrupt
  <idle>-0       0d...   141.555327: function:                native_apic_mem_write
  <idle>-0       0d...   141.555327: function:                exit_idle
  <idle>-0       0d...   141.555327: function:                irq_enter
  <idle>-0       0d...   141.555327: function:                   rcu_irq_enter
  <idle>-0       0d...   141.555328: function:                   idle_cpu
  <idle>-0       0d.h.   141.555328: function:                   tick_check_idle
  <idle>-0       0d.h.   141.555328: function:                      tick_check_oneshot_broadcast
  <idle>-0       0d.h.   141.555328: function:                      ktime_get

Notice that we traced smp_apic_timer_interrupt, native_apic_mem_write,
exit_idle, irq_enter, and rcu_irq_enter, before rcu even was informed
that we are coming out of idle.

Then idle_cpu was also traced before the preempt_count was changed to
notify that we are in an interrupt (the 'h' in "0d.h.").

-- Steve