Re: Perf: event wakeup discards sched_waking events

From: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To: rostedt <rostedt@goodmis.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	linux-kernel <linux-kernel@vger.kernel.org>,
	Thomas Gleixner <tglx@linutronix.de>
Subject: Re: Perf: event wakeup discards sched_waking events
Date: Mon, 14 Jan 2019 17:31:09 -0500 (EST)	[thread overview]
Message-ID: <1546659617.746.1547505069810.JavaMail.zimbra@efficios.com> (raw)
In-Reply-To: <20190114170403.344f7247@gandalf.local.home>

----- On Jan 14, 2019, at 5:04 PM, rostedt rostedt@goodmis.org wrote:

> On Mon, 14 Jan 2019 16:36:59 -0500 (EST)
> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> 
>> Can ftrace end up in the same situation through rb_wake_up_waiters() ? I suspect
>> the tail recursion would be hard to trigger if the wakeup only happens once per
>> page though, unless the events generated end up filling up a page.
> 
> And only events from the irq work that was caused by the ftrace wakeup,
> which is highly unlikely.
> 
> Note, the lastest kernel only wakes up after half the buffer is full
> (by default, but that can be changed), as I found that it gives the
> best performance to keeping up with traces. I can actually trace small
> loads and get all events now. Before, the waking of the tracer would
> cause its own events to fill up the buffer and not be able to keep up
> even on simple loads.
> 
>> 
>> FWIW, LTTng avoids this entire issue by using a timer-based polling mechanism
>> to ensure the tracing code does not call into the scheduler wakeup.
> 
> Does your timer stop if the system is idle and tracing is enabled?

LTTng exposes three functions for integration with CONFIG_NO_HZ:

lib_ring_buffer_tick_nohz_flush()
lib_ring_buffer_tick_nohz_stop()
lib_ring_buffer_tick_nohz_restart()

Unfortunately, last time I checked, the Linux kernel did not expose any
modules API to register nohz notifiers to call those functions when going
to idle and getting out of idle. Therefore, the plumbing is there
in LTTng (it has been done when I implemented libringbuffer about 6 years
ago intending to contribute it to the Linux kernel), but this code has
never been reachable within lttng-modules due to lack of kernel support.

Thomas Gleixner expressed strong feelings against introducing tick nohz
notifiers for kernel modules in the past, so I never pushed in that
direction.

Perhaps there is now a way to properly wire this up that I've missed ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com