From: Namhyung Kim <namhyung@kernel.org>
To: Kalesh Singh <kaleshsingh@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>,
Hridya Valsaraju <hridya@google.com>,
Jonathan Corbet <corbet@lwn.net>,
Steven Rostedt <rostedt@goodmis.org>,
Ingo Molnar <mingo@redhat.com>, Shuah Khan <shuah@kernel.org>,
Tom Zanussi <zanussi@kernel.org>,
Masami Hiramatsu <mhiramat@kernel.org>,
"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
"open list:KERNEL SELFTEST FRAMEWORK"
<linux-kselftest@vger.kernel.org>
Subject: Re: [PATCH 0/5] tracing: Extend histogram triggers expression parsing
Date: Thu, 30 Sep 2021 15:58:06 -0700 [thread overview]
Message-ID: <CAM9d7ciQC1sV8hOOsgHVLxk7sze_Qp_dBqBkK_FrtVhZ0=AzZQ@mail.gmail.com> (raw)
In-Reply-To: <CAC_TJvdv7sT-FmD1S-ZHnpAGvFR=1WBc6jEKBm+q5Wpp6S34PQ@mail.gmail.com>
Hi Kalesh,
On Wed, Sep 15, 2021 at 1:09 PM Kalesh Singh <kaleshsingh@google.com> wrote:
>
> On Wed, Sep 15, 2021 at 12:53 PM Kalesh Singh <kaleshsingh@google.com> wrote:
> >
> > The frequency of the rss_stat trace event is known to be of the same
> > magnitude as that of the sched_switch event on Android devices. This can
> > cause flooding of the trace buffer with rss_stat traces leading to a
> > decreased trace buffer capacity and loss of data.
> >
> > If it is not necessary to monitor very small changes in rss (as is the
> > case in Android) then the rss_stat tracepoint can be throttled to only
> > emit the event once there is a large enough change in the rss size.
> > The original patch that introduced the rss_stat tracepoint also proposed
> > a fixed throttling mechanism that only emits the rss_stat event
> > when the rss size crosses a 512KB boundary. It was concluded that more
> > generic support for this type of filtering/throttling was need, so that
> > it can be applied to any trace event. [1]
> >
> > From the discussion in [1], histogram triggers seemed the most likely
> > candidate to support this type of throttling. For instance to achieve the
> > same throttling as was proposed in [1]:
> >
> > (1) Create a histogram variable to save the 512KB bucket of the rss size
> > (2) Use the onchange handler to generate a synthetic event when the
> > rss size bucket changes.
> >
> > The only missing pieces to support such a hist trigger are:
> > (1) Support for setting a hist variable to a specific value -- to set
> > the bucket size / granularity.
> > (2) Support for division arithmetic operation -- to determine the
> > corresponding bucket for an rss size.
> >
> > This series extends histogram trigger expressions to:
> > (1) Allow assigning numeric literals to hist variable (eg. x=1234)
> > and using literals directly in expressions (eg. x=size/1234)
> > (2) Support division and multiplication in hist expressions.
> > (eg. a=$x/$y*z); and
> > (3) Fixes expression parsing for non-associative operators: subtraction
> > and division. (eg. 8-4-2 should be 2 not 6)
> >
> > The rss_stat event can then be throttled using histogram triggers as
> > below:
> >
> > # Create a synthetic event to monitor instead of the high frequency
> > # rss_stat event
> > echo 'rss_stat_throttled unsigned int mm_id; unsigned int curr;
> > int member; long size' >> tracing/synthetic_events
> >
> > # Create a hist trigger that emits the synthetic rss_stat_throttled
> > # event only when the rss size crosses a 512KB boundary.
> > echo 'hist:keys=common_pid:bucket=size/0x80000:onchange($bucket)
> > .rss_stat_throttled(mm_id,curr,member,size)'
> > >> events/kmem/rss_stat/trigger
> >
>
> Sorry, I have a clerical mistake here. The above key should be:
> s/keys=common_pid/keys=keys=mm_id,member
>
> The rss size is specific to the mm struct's member not the pid.
> The results below were captured with the correct key so no changes there.
>
> > ------ Test Results ------
> > Histograms can also be used to evaluate the effectiveness of this
> > throttling by noting the Total Hits on each trigger:
> >
> > echo 'hist:keys=common_pid' >> events/sched/sched_switch/trigger
> > echo 'hist:keys=common_pid' >> events/kmem/rss_stat/trigger
> > echo 'hist:keys=common_pid'
> > >> events/synthetic/rss_stat_throttled/trigger
> >
> > Allowing the above example (512KB granularity) run for 5 minutes on
> > an arm64 device with 5.10 kernel:
> >
> > sched_switch : total hits = 147153
> > rss_stat : total hits = 38863
> > rss_stat_throttled: total hits = 2409
> >
> > The synthetic rss_stat_throttled event is ~16x less frequent than the
> > rss_stat event when using a 512KB granularity.
> >
> >
> > The results are more pronounced when rss size is changing at a higher
> > rate in small increments. For instance the following results were obtained
> > by recording the hits on the above events for a run of Android's
> > lmkd_unit_test [2], which continually forks processes that map anonymous
> > memory until there is an oom kill:
> >
> > sched_switch : total hits = 148832
> > rss_stat : total hits = 4754802
> > rss_stat_throttled: total hits = 96214
> >
> > In this stress this, the synthetic rss_stat_throttled event is ~50x less
> > frequent than the rss_stat event when using a 512KB granularity.
> >
> >
> > [1] https://lore.kernel.org/lkml/20190903200905.198642-1-joel@joelfernandes.org/
> > [2] https://cs.android.com/android/platform/superproject/+/master:system/memory/lmkd/tests/lmkd_test.cpp
> >
> > Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
Reviewed-by: Namhyung Kim <namhyung@kernel.org>
Thanks,
Namhyung
> >
> > Kalesh Singh (5):
> > tracing: Add support for creating hist trigger variables from literal
> > tracing: Add division and multiplication support for hist triggers
> > tracing: Fix operator precedence for hist triggers expression
> > tracing/selftests: Add tests for hist trigger expression parsing
> > tracing/histogram: Document expression arithmetic and constants
> >
> > Documentation/trace/histogram.rst | 14 +
> > kernel/trace/trace_events_hist.c | 318 +++++++++++++++---
> > .../testing/selftests/ftrace/test.d/functions | 4 +-
> > .../trigger/trigger-hist-expressions.tc | 73 ++++
> > 4 files changed, 357 insertions(+), 52 deletions(-)
> > create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-expressions.tc
> >
> >
> > base-commit: 3ca706c189db861b2ca2019a0901b94050ca49d8
> > --
> > 2.33.0.309.g3052b89438-goog
> >
next prev parent reply other threads:[~2021-09-30 22:58 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-15 19:52 [PATCH 0/5] tracing: Extend histogram triggers expression parsing Kalesh Singh
2021-09-15 19:52 ` [PATCH 1/5] tracing: Add support for creating hist trigger variables from literal Kalesh Singh
2021-10-19 20:48 ` Steven Rostedt
2021-10-19 21:21 ` Kalesh Singh
2021-10-19 21:26 ` Steven Rostedt
2021-10-19 21:49 ` Kalesh Singh
2021-09-15 19:52 ` [PATCH 2/5] tracing: Add division and multiplication support for hist triggers Kalesh Singh
2021-10-19 20:45 ` Steven Rostedt
2021-10-19 21:24 ` Kalesh Singh
2021-09-15 19:52 ` [PATCH 3/5] tracing: Fix operator precedence for hist triggers expression Kalesh Singh
2021-09-15 19:52 ` [PATCH 4/5] tracing/selftests: Add tests for hist trigger expression parsing Kalesh Singh
2021-09-15 19:52 ` [PATCH 5/5] tracing/histogram: Document expression arithmetic and constants Kalesh Singh
2021-09-15 20:09 ` [PATCH 0/5] tracing: Extend histogram triggers expression parsing Kalesh Singh
2021-09-30 22:58 ` Namhyung Kim [this message]
2021-10-02 0:54 ` Kalesh Singh
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAM9d7ciQC1sV8hOOsgHVLxk7sze_Qp_dBqBkK_FrtVhZ0=AzZQ@mail.gmail.com' \
--to=namhyung@kernel.org \
--cc=corbet@lwn.net \
--cc=hridya@google.com \
--cc=kaleshsingh@google.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=mingo@redhat.com \
--cc=rostedt@goodmis.org \
--cc=shuah@kernel.org \
--cc=surenb@google.com \
--cc=zanussi@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).