Re: [RFC v1 00/14] Implement call_rcu_lazy() and miscellaneous fixes

[Joel Fernandes <joel@joelfernandes.org>]

On Fri, May 13, 2022 at 9:36 AM Uladzislau Rezki <urezki@gmail.com> wrote:
>
> > > On Thu, May 12, 2022 at 10:37 AM Uladzislau Rezki <urezki@gmail.com> wrote:
> > > >
> > > > > On Thu, May 12, 2022 at 03:56:37PM +0200, Uladzislau Rezki wrote:
> > > > > > Never mind. I port it into 5.10
> > > > >
> > > > > Oh, this is on mainline. Sorry about that. If you want I have a tree here for
> > > > > 5.10 , although that does not have the kfree changes, everything else is
> > > > > ditto.
> > > > > https://github.com/joelagnel/linux-kernel/tree/rcu-nocb-4
> > > > >
> > > > No problem. kfree_rcu two patches are not so important in this series.
> > > > So i have backported them into my 5.10 kernel because the latest kernel
> > > > is not so easy to up and run on my device :)
> > >
> > > Actually I was going to write here, apparently some tests are showing
> > > kfree_rcu()->call_rcu_lazy() causing possible regression. So it is
> > > good to drop those for your initial testing!
> > >
> > Yep, i dropped both. The one that make use of call_rcu_lazy() seems not
> > so important for kfree_rcu() because we do batch requests there anyway.
> > One thing that i would like to improve in kfree_rcu() is a better utilization
> > of page slots.
> >
> > I will share my results either tomorrow or on Monday. I hope that is fine.
> >
>
> Here we go with some data on our Android handset that runs 5.10 kernel. The test
> case i have checked was a "static image" use case. Condition is: screen ON with
> disabled all connectivity.
>
> 1.
> First data i took is how many wakeups cause an RCU subsystem during this test case
> when everything is pretty idling. Duration is 360 seconds:
>
> <snip>
> serezkiul@seldlx26095:~/data/call_rcu_lazy$ ./psp ./perf_360_sec_rcu_lazy_off.script | sort -nk 6 | grep rcu

Nice! Do you mind sharing this script? I was just talking to Rushikesh
that we want something like this during testing. Appreciate it. Also,
if we dump timer wakeup reasons/callbacks that would also be awesome.

FWIW, I wrote a BPF tool that periodically dumps callbacks and can
share that with you on request as well. That is probably not in a
shape for mainline though (Makefile missing and such).

> name:                         rcub/0 pid:         16 woken-up     2     interval: min 86772734  max 86772734    avg 43386367
> name:                        rcuop/7 pid:         69 woken-up     4     interval: min  4189     max  8050       avg  5049
> name:                        rcuop/6 pid:         62 woken-up    55     interval: min  6910     max 42592159    avg 3818752
[..]

> There is a big improvement in lazy case. Number of wake-ups got reduced quite a lot
> and it is really good!

Cool!

> 2.
> Please find in attachment two power plots. The same test case. One is related to a
> regular use of call_rcu() and second one is "lazy" usage. There is light a difference
> in power, it is ~2mA. Event though it is rather small but it is detectable and solid
> what is also important, thus it proofs the concept. Please note it might be more power
> efficient for other arches and platforms. Because of different HW design that is related
> to C-states of CPU and energy that is needed to in/out of those deep power states.

Nice! I wonder if you still have other frequent callbacks on your
system that are getting queued during the tests. Could you dump the
rcu_callbacks trace event and see if you have any CBs frequently
called that the series did not address?

Also, one more thing I was curious about is - do you see savings when
you pin the rcu threads to the LITTLE CPUs of the system? The theory
being, not disturbing the BIG CPUs which are more power hungry may let
them go into a deeper idle state and save power (due to leakage
current and so forth).

> So a front-lazy-batching is something worth to have, IMHO :)

Exciting! Being lazy pays off some times ;-) ;-). If you are Ok with
it, we can add your data to the LPC slides as well about your
investigation (with attribution to you).

thanks,

 - Joel