From: Tom Zanussi <zanussi@kernel.org>
To: rostedt@goodmis.org
Cc: tglx@linutronix.de, mhiramat@kernel.org, namhyung@kernel.org,
vedang.patel@intel.com, bigeasy@linutronix.de,
joel@joelfernandes.org, mathieu.desnoyers@efficios.com,
julia@ni.com, linux-kernel@vger.kernel.org,
linux-rt-users@vger.kernel.org
Subject: [PATCH v7 11/16] tracing: Add hist trigger onchange() handler Documentation
Date: Wed, 14 Nov 2018 14:18:08 -0600 [thread overview]
Message-ID: <e73e3c65facffedf2f46be48ea386092f53c14fd.1542221863.git.tom.zanussi@linux.intel.com> (raw)
In-Reply-To: <cover.1542221862.git.tom.zanussi@linux.intel.com>
In-Reply-To: <cover.1542221862.git.tom.zanussi@linux.intel.com>
From: Tom Zanussi <tom.zanussi@linux.intel.com>
Add Documentation for the hist:onchange($var) handler.
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
Documentation/trace/histogram.rst | 98 +++++++++++++++++++++++++++++++++++++++
1 file changed, 98 insertions(+)
diff --git a/Documentation/trace/histogram.rst b/Documentation/trace/histogram.rst
index 353317bc3825..79476c906b1a 100644
--- a/Documentation/trace/histogram.rst
+++ b/Documentation/trace/histogram.rst
@@ -1869,6 +1869,7 @@ The available handlers are:
- onmatch(matching.event) - invoke action on any addition or update
- onmax(var) - invoke action if var exceeds current max
+ - onchange(var) - invoke action if var changes
The available actions are:
@@ -2140,6 +2141,103 @@ The following commonly-used handler.action pairs are available:
<...>-2102 [005] d..4 309.875185: sched_wake_idle_without_ipi: cpu=1
<idle>-0 [001] d..3 309.875200: sched_switch: prev_comm=swapper/1 prev_pid=0 prev_prio=120 prev_state=S ==> next_comm=cyclictest next_pid=2103 next_prio=19
+ - onchange(var).save(field,.. .)
+
+ The 'onchange(var).save(field,...)' hist trigger action is invoked
+ whenever the value of 'var' associated with a histogram entry
+ changes.
+
+ The end result is that the trace event fields specified as the
+ onchange.save() params will be saved if 'var' changes for that
+ hist trigger entry. This allows context from the event that
+ changed the value to be saved for later reference. When the
+ histogram is displayed, additional fields displaying the saved
+ values will be printed.
+
+ - onchange(var).snapshot()
+
+ The 'onchange(var).snapshot()' hist trigger action is invoked
+ whenever the value of 'var' associated with a histogram entry
+ changes.
+
+ The end result is that a global snapshot of the trace buffer will
+ be saved in the tracing/snapshot file if 'var' changes for any
+ hist trigger entry.
+
+ Note that in this case the changed value is a global variable
+ associated withe current trace instance. The key of the specific
+ trace event that caused the value to change and the global value
+ itself are displayed, along with a message stating that a snapshot
+ has been taken and where to find it. The user can use the key
+ information displayed to locate the corresponding bucket in the
+ histogram for even more detail.
+
+ As an example the below defines a hist trigger on the tcp_probe
+ event, keyed on dport. Whenever a tcp_probe event occurs, the
+ cwnd field is checked against the current value stored in the
+ $cwnd variable. If the value has changed, a snapshot is taken.
+ As part of the setup, all the scheduler and tcp events are also
+ enabled, which are the events that will show up in the snapshot
+ when it is taken at some point:
+
+ # echo 1 > /sys/kernel/debug/tracing/events/sched/enable
+ # echo 1 > /sys/kernel/debug/tracing/events/tcp/enable
+
+ # echo 'hist:keys=dport:cwnd=snd_cwnd: \
+ onchange($cwnd).save(snd_wnd,srtt,rcv_wnd): \
+ onchange($cwnd).snapshot()' >> \
+ /sys/kernel/debug/tracing/events/tcp/tcp_probe/trigger
+
+ When the histogram is displayed, for each bucket the tracked value
+ and the saved values corresponding to that value are displayed
+ following the rest of the fields.
+
+ If a snaphot was taken, there is also a message indicating that,
+ along with the value and event that triggered the snapshot:
+
+ # cat /sys/kernel/debug/tracing/events/tcp/tcp_probe/hist
+ { dport: 1521 } hitcount: 8
+ changed: 10 snd_wnd: 35456 srtt: 154262 rcv_wnd: 42112
+
+ { dport: 80 } hitcount: 23
+ changed: 10 snd_wnd: 28960 srtt: 19604 rcv_wnd: 29312
+
+ { dport: 9001 } hitcount: 172
+ changed: 10 snd_wnd: 48384 srtt: 260444 rcv_wnd: 55168
+
+ { dport: 443 } hitcount: 211
+ changed: 10 snd_wnd: 26960 srtt: 17379 rcv_wnd: 28800
+
+ Snapshot taken (see tracing/snapshot). Details:
+ triggering value { onchange($cwnd) }: 10
+ triggered by event with key: { dport: 80 }
+
+ Totals:
+ Hits: 414
+ Entries: 4
+ Dropped: 0
+
+ In the above case, the event that triggered the snapshot has the
+ key with dport == 80. If you look at the bucket that has 80 as
+ the key, you'll find the additional values save()'d along with the
+ changed value for that bucket, which should be the same as the
+ global changed value (since that was the same value that triggered
+ the global snapshot).
+
+ And finally, looking at the snapshot data should show at or near
+ the end the event that triggered the snapshot:
+
+ # cat /sys/kernel/debug/tracing/snapshot
+
+ gnome-shell-1261 [006] dN.3 49.823113: sched_stat_runtime: comm=gnome-shell pid=1261 runtime=49347 [ns] vruntime=1835730389 [ns]
+ kworker/u16:4-773 [003] d..3 49.823114: sched_switch: prev_comm=kworker/u16:4 prev_pid=773 prev_prio=120 prev_state=R+ ==> next_comm=kworker/3:2 next_pid=135 next_prio=120
+ gnome-shell-1261 [006] d..3 49.823114: sched_switch: prev_comm=gnome-shell prev_pid=1261 prev_prio=120 prev_state=R+ ==> next_comm=kworker/6:2 next_pid=387 next_prio=120
+ kworker/3:2-135 [003] d..3 49.823118: sched_stat_runtime: comm=kworker/3:2 pid=135 runtime=5339 [ns] vruntime=17815800388 [ns]
+ kworker/6:2-387 [006] d..3 49.823120: sched_stat_runtime: comm=kworker/6:2 pid=387 runtime=9594 [ns] vruntime=14589605367 [ns]
+ kworker/6:2-387 [006] d..3 49.823122: sched_switch: prev_comm=kworker/6:2 prev_pid=387 prev_prio=120 prev_state=R+ ==> next_comm=gnome-shell next_pid=1261 next_prio=120
+ kworker/3:2-135 [003] d..3 49.823123: sched_switch: prev_comm=kworker/3:2 prev_pid=135 prev_prio=120 prev_state=T ==> next_comm=swapper/3 next_pid=0 next_prio=120
+ <idle>-0 [004] ..s7 49.823798: tcp_probe: src=10.0.0.10:54326 dest=23.215.104.193:80 mark=0x0 length=32 snd_nxt=0xe3ae2ff5 snd_una=0xe3ae2ecd snd_cwnd=10 ssthresh=2147483647 snd_wnd=28960 srtt=19604 rcv_wnd=29312
+
3. User space creating a trigger
--------------------------------
--
2.14.1
next prev parent reply other threads:[~2018-11-14 20:19 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-14 20:17 [PATCH v7 00/16] tracing: Hist trigger snapshot and onchange additions Tom Zanussi
2018-11-14 20:17 ` [PATCH v7 01/16] tracing: Refactor hist trigger action code Tom Zanussi
2018-11-14 20:17 ` [PATCH v7 02/16] tracing: Make hist trigger Documentation better reflect actions/handlers Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 03/16] tracing: Add hist trigger handler.action documentation to README Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 04/16] tracing: Split up onmatch action data Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 05/16] tracing: Generalize hist trigger onmax and save action Tom Zanussi
2018-11-23 2:50 ` Namhyung Kim
2018-12-03 22:22 ` Tom Zanussi
2018-12-04 7:25 ` Namhyung Kim
2018-12-04 19:53 ` Tom Zanussi
2018-11-23 7:01 ` Namhyung Kim
2018-11-27 22:48 ` Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 06/16] tracing: Add conditional snapshot Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 07/16] tracing: Add hist trigger snapshot() action Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 08/16] tracing: Add hist trigger snapshot() action Documentation Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 09/16] tracing: Add hist trigger snapshot() action test case Tom Zanussi
2018-11-26 13:03 ` Masami Hiramatsu
2018-11-27 22:53 ` Tom Zanussi
2018-11-28 2:15 ` Masami Hiramatsu
2018-11-29 1:12 ` Tom Zanussi
2018-12-04 19:59 ` Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 10/16] tracing: Add hist trigger onchange() handler Tom Zanussi
2018-11-14 20:18 ` Tom Zanussi [this message]
2018-11-14 20:18 ` [PATCH v7 12/16] tracing: Add hist trigger onchange() handler test case Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 13/16] tracing: Add alternative synthetic event trace action syntax Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 14/16] tracing: Add alternative synthetic event trace action test case Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 15/16] tracing: Add hist trigger action 'expected fail' " Tom Zanussi
2018-11-14 20:18 ` [PATCH v7 16/16] tracing: Add SPDX license GPL-2.0 license identifier to inter-event testcases Tom Zanussi
2018-11-26 14:09 ` [PATCH v7 00/16] tracing: Hist trigger snapshot and onchange additions Masami Hiramatsu
2018-11-26 21:21 ` Tom Zanussi
2018-11-29 13:52 ` Masami Hiramatsu
2018-11-29 14:54 ` Masami Hiramatsu
2018-11-29 15:07 ` [PATCH] sefltests/ftrace: Use /bin/echo for output with options Masami Hiramatsu
2018-11-29 16:32 ` Steven Rostedt
2018-11-29 16:43 ` Tom Zanussi
2018-11-29 16:49 ` Steven Rostedt
2019-03-04 20:09 ` Tom Zanussi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=e73e3c65facffedf2f46be48ea386092f53c14fd.1542221863.git.tom.zanussi@linux.intel.com \
--to=zanussi@kernel.org \
--cc=bigeasy@linutronix.de \
--cc=joel@joelfernandes.org \
--cc=julia@ni.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=mhiramat@kernel.org \
--cc=namhyung@kernel.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=vedang.patel@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).