Re: [RFC v4 3/4] irqflags: Avoid unnecessary calls to trace_ if you can

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

----- On Apr 24, 2018, at 2:59 PM, Joel Fernandes joelaf@google.com wrote:

> Hi Paul,
> 
> On Tue, Apr 24, 2018 at 11:26 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
>> On Tue, Apr 24, 2018 at 11:23:02AM -0700, Paul E. McKenney wrote:
>>> On Tue, Apr 24, 2018 at 10:26:58AM -0700, Paul E. McKenney wrote:
>>> > On Tue, Apr 24, 2018 at 09:01:34AM -0700, Joel Fernandes wrote:
>>> > > On Tue, Apr 24, 2018 at 8:56 AM, Paul E. McKenney
>>> > > <paulmck@linux.vnet.ibm.com> wrote:
>>> > > > On Mon, Apr 23, 2018 at 05:22:44PM -0400, Steven Rostedt wrote:
>>> > > >> On Mon, 23 Apr 2018 13:12:21 -0400 (EDT)
>>> > > >> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
>>> > > >>
>>> > > >>
>>> > > >> > I'm inclined to explicitly declare the tracepoints with their given
>>> > > >> > synchronization method. Tracepoint probe callback functions for currently
>>> > > >> > existing tracepoints expect to have preemption disabled when invoked.
>>> > > >> > This assumption will not be true anymore for srcu-tracepoints.
>>> > > >>
>>> > > >> Actually, why not have a flag attached to the tracepoint_func that
>>> > > >> states if it expects preemption to be enabled or not? If a
>>> > > >> trace_##event##_srcu() is called, then simply disable preemption before
>>> > > >> calling the callbacks for it. That way if a callback is fine for use
>>> > > >> with srcu, then it would require calling
>>> > > >>
>>> > > >>       register_trace_##event##_may_sleep();
>>> > > >>
>>> > > >> Then if someone uses this on a tracepoint where preemption is disabled,
>>> > > >> we simply do not call it.
>>> > > >
>>> > > > One more stupid question...  If we are having to trace so much stuff
>>> > > > in the idle loop, are we perhaps grossly overstating the extent of that
>>> > > > "idle" loop?  For being called "idle", this code seems quite busy!
>>> > >
>>> > > ;-)
>>> > > The performance hit I am observing is when running a heavy workload,
>>> > > like hackbench or something like that. That's what I am trying to
>>> > > correct.
>>> > > By the way is there any limitation on using SRCU too early during
>>> > > boot? I backported Mathieu's srcu tracepoint patches but the kernel
>>> > > hangs pretty early in the boot. I register lockdep probes in
>>> > > start_kernel. I am hoping that's not why.
>>> > >
>>> > > I could also have just screwed up the backporting... may be for my
>>> > > testing, I will just replace the rcu API with the srcu instead of all
>>> > > of Mathieu's new TRACE_EVENT macros for SRCU, since all I am trying to
>>> > > do right now is measure the performance of my patches with SRCU.
>>> >
>>> > Gah, yes, there is an entry on my capacious todo list on making SRCU
>>> > grace periods work during early boot and mid-boot.  Let me see what
>>> > I can do...
>>>
>>> OK, just need to verify that you are OK with call_srcu()'s callbacks
>>> not being invoked until sometime during core_initcall() time.  (If you
>>> really do need them to be invoked before that, in theory it is possible,
>>> but in practice it is weird, even for RCU.)
>>
>> Oh, and that early at boot, you will need to use DEFINE_SRCU() or
>> DEFINE_STATIC_SRCU() rather than dynamic allocation and initialization.
>>
>>                                                         Thanx, Paul
>>
> 
> Oh ok.
> 
> About call_rcu, calling it later may be an issue since we register the
> probes in start_kernel, for the first probe call_rcu will be sched,
> but for the second one I think it'll try to call_rcu to get rid of the
> first one.
> 
> This is the relevant code that gets called when probes are added:
> 
> static inline void release_probes(struct tracepoint_func *old)
> {
>        if (old) {
>                struct tp_probes *tp_probes = container_of(old,
>                        struct tp_probes, probes[0]);
>                call_rcu_sched(&tp_probes->rcu, rcu_free_old_probes);
>        }
> }
> 
> Maybe we can somehow defer the call_srcu until later? Would that be possible?
> 
> also Mathieu, you didn't modify the call_rcu_sched in your prototype
> to be changed to use call_srcu, should you be doing that?

You're right, I think I should have introduced a call_srcu in there.
It's missing in my prototype.

However, in the prototype I did, we need to wait for *both* sched-rcu
and SRCU grace periods, because we don't track which site is using which
rcu flavor.

So you could achieve this relatively easily by means of two chained
RCU callbacks, e.g.:

release_probes() calls call_rcu_sched(... , rcu_free_old_probes)

and then in rcu_free_old_probes() do:

call_srcu(... , srcu_free_old_probes)

and perform kfree(container_of(head, struct tp_probes, rcu));
within srcu_free_old_probes.

It is somewhat a hack, but should work.

Thanks,

Mathieu

> 
> thanks,
> 
>  - Joel

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com