From: Frederic Weisbecker <fweisbec@gmail.com>
To: Ingo Molnar <mingo@elte.hu>
Cc: LKML <linux-kernel@vger.kernel.org>,
Li Zefan <lizf@cn.fujitsu.com>,
Frederic Weisbecker <fweisbec@gmail.com>,
Zhao Lei <zhaolei@cn.fujitsu.com>,
Steven Rostedt <rostedt@goodmis.org>,
Tom Zanussi <tzanussi@gmail.com>,
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>,
Oleg Nesterov <oleg@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>
Subject: [PATCH 19/19] tracing/workqueue: provide documentation for the workqueue tracer
Date: Thu, 30 Apr 2009 02:27:20 +0200 [thread overview]
Message-ID: <1241051240-4280-20-git-send-email-fweisbec@gmail.com> (raw)
In-Reply-To: <1241051240-4280-1-git-send-email-fweisbec@gmail.com>
Provide a quick documentation for both stat and event tracing with
the workqueue tracer.
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Zhao Lei <zhaolei@cn.fujitsu.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Tom Zanussi <tzanussi@gmail.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
Documentation/trace/workqueue.txt | 114 +++++++++++++++++++++++++++++++++++++
1 files changed, 114 insertions(+), 0 deletions(-)
diff --git a/Documentation/trace/workqueue.txt b/Documentation/trace/workqueue.txt
new file mode 100644
index 0000000..4019a8a
--- /dev/null
+++ b/Documentation/trace/workqueue.txt
@@ -0,0 +1,114 @@
+The workqueue tracer is a tool that can provide informations
+about worklets execution frequency, latency and worklets/workqueues
+lifecycle.
+
+== A statistical/histogram tracer ==
+
+This tracer provides a snapshot of the workqueues and worklets
+current state.
+
+Eg:
+
+# CPU INSERTED EXECUTED MAX us AVG us TASKNAME:PID
+# | | | | ` | -WORKFUNC
+# | | | | | |
+ 0 5903 5977 events/0:9
+ 0 1 1 38 38 |-hpet_work
+ 0 49 48 295 224 |-rt_worker_func
+ 0 2934 2933 73 11 |-vmstat_update
+ 0 935 935 13143 264 |-console_callback
+ 0 80 80 1634632587 45804402 |-test_work
+ 0 10 9 239 85 |-rekey_seq_generator
+ 0 10 10 0 0 |-lru_add_drain_per_cpu
+ 0 1 1 4 4 |-sync_cmos_clock
+ 0 651 651 366 65 |-flush_to_ldisc
+ 0 1105 1105 13171 256 |-fb_flashcursor
+ 0 127 120 213 18 `-dst_gc_task
+ 0 0 0 cpuset:11
+ 0 1409 1409 khelper:12
+ 0 1409 1409 124030 754 `-__call_usermodehelper
+ 0 0 0 kintegrityd/0:174
+ 0 5823 5823 kblockd/0:179
+ 0 336 336 176 40 |-blk_unplug_work
+ 0 5487 5487 141 21 `-cfq_kick_queue
+ 0 0 0 kacpid:186
+ 0 0 0 kacpi_notify:187
+ 0 0 0 tifm:498
+ 0 19505 19505 ata/0:508
+ 0 19505 19505 114 24 `-ata_pio_task
+ 0 0 0 ata_aux:511
+ 0 0 0 aio/0:871
+ 0 0 0 crypto/0:897
+ 0 0 0 scsi_tgtd/0:2368
+ 0 0 0 iscsi_eh:2386
+ 0 0 0 kpsmoused:2525
+ 0 0 0 hd-audio0:2714
+ 0 0 0 krxrpcd/0:2752
+ 0 242 242 reiserfs/0:2896
+ 0 242 242 4563280 21918 `-flush_async_commits
+ 1 0 0 reiserfs/1:6438
+ 1 0 0 krxrpcd/1:6439
+ 1 0 0 scsi_tgtd/1:6440
+ 1 0 0 crypto/1:6441
+ 1 0 0 aio/1:6442
+ 1 62 62 ata/1:6443
+ 1 62 62 28 24 `-ata_pio_task
+ 1 9 9 kblockd/1:6444
+ 1 9 9 102 54 `-blk_unplug_work
+ 1 0 0 kintegrityd/1:6445
+ 1 2544 2545 events/1:6446
+ 1 2541 2540 22 11 |-vmstat_update
+ 1 1 1 15 15 |-hpet_work
+ 1 2 2 10 6 `-lru_add_drain_per_cpu
+
+For each worklet, you can find how much it has been enqueued
+and executed. Also the MAX us and AVG us give you respectively
+the maximum and average time of execution in microseconds
+for a given worklet.
+
+A series of worklets is grouped under a workqueue using the
+following pattern:
+
+cpu_workqueue_thread[/cpu_nr]:pid
+|-work1
+|-work2
+...
+`-last_work
+
+If cpu_nr is not given, then the workqueue thread is
+a singlethread workqueue, bound to every possible cpu.
+
+Use the following commands to read a snapshot:
+
+mount -t debugfs debug /sys/kernel/debug/
+cat /sys/kernel/debug/tracing/trace_stat/workqueues
+
+
+== An event tracer ==
+
+The workqueue tracer also uses a set of events tracepoints which
+can be used to trace each key event from every workqueues or worklets.
+
+You can refer to Documentation/trace/events.txt for a detailed
+use of the events tracepoints. How to individually toggle them, how
+to use filters, which field is what, etc...
+
+For a simple capture of all workqueues events:
+
+# mount -t debugfs debug /sys/kernel/debug/
+# cd /sys/kernel/debug/tracing/events/workqueue
+# ls
+filter worklet_complete worklet_enqueue_delayed workqueue_creation workqueue_flush
+worklet_cancel worklet_enqueue worklet_execute workqueue_destruction
+# for e in $(ls); do echo 1 > $e/enable; done
+# cat /sys/kernel/debug/tracing/trace
+
+ #
+ # TASK-PID CPU# TIMESTAMP FUNCTION
+ # | | | | |
+
+<...>-5627 [000] 4597.858645: worklet_enqueue: thread=ata/0:508 func=ata_pio_task+0x0/0x280 cpu=0
+<...>-508 [000] 4597.858660: worklet_execute: thread=ata/0:508 work=ffff88007e9e67b8 func=ata_pio_task+0x0/0x280
+<...>-508 [000] 4597.858684: worklet_complete: thread=ata/0:508 work=ffff88007e9e67b8
+<...>-2437 [000] 4597.861259: worklet_cancel: func=ata_pio_task+0x0/0x280
+[...]
--
1.6.2.3
next prev parent reply other threads:[~2009-04-30 0:34 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-04-30 0:27 [PATCH 00/19] tracing/workqueue: events support/enhancements, worklets tracing, fixes Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 01/19] ftrace, workqueuetrace: make workqueue tracepoints use TRACE_EVENT macro Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 02/19] trace_workqueue: use list_for_each_entry() instead of list_for_each_entry_safe() Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 03/19] trace_workqueue: remove cpu_workqueue_stats->first_entry Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 04/19] trace_workqueue: remove blank line between each cpu Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 05/19] trace_workqueue: add worklet information Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 06/19] tracing/workqueue: turn cpu_workqueue_stats::inserted into unsigned int Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 07/19] workqueue_tracepoint: introduce workqueue_handler_exit tracepoint and rename workqueue_execution to workqueue_handler_entry Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 08/19] workqueue_tracepoint: add workqueue_flush and worklet_cancel tracepoint Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 09/19] workqueue_tracepoint: change tracepoint name to fit worklet and workqueue lifecycle Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 10/19] workqueue_trace: separate worklet_insertion into worklet_enqueue and worklet_enqueue_delayed Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 11/19] tracing/workqueue: turn workfunc_stats::inserted into unsigned int Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 12/19] tracing/workqueue: avoid accessing task_struct's member variable in stat file read Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 13/19] tracing/workqueue: use list_for_each_entry instead of list_for_each_entry_safe in probe_workqueue_destruction() Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 14/19] tracing/workqueue: add max execution time mesurement for per worklet Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 15/19] tracing/workqueue: add execution time average column Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 16/19] tracing/workqueue: add open/release file handlers Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 17/19] tracing/workqueue: defer workqueue stat release if needed Frederic Weisbecker
2009-04-30 0:27 ` [PATCH 18/19] tracing/workqueue: use the original cpu affinity on probe_workqueue_destruction Frederic Weisbecker
2009-04-30 0:27 ` Frederic Weisbecker [this message]
2009-04-30 1:19 ` [PATCH 19/19] tracing/workqueue: provide documentation for the workqueue tracer KOSAKI Motohiro
2009-04-30 8:45 ` [PATCH 00/19] tracing/workqueue: events support/enhancements, worklets tracing, fixes Ingo Molnar
2009-04-30 12:48 ` Frederic Weisbecker
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1241051240-4280-20-git-send-email-fweisbec@gmail.com \
--to=fweisbec@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lizf@cn.fujitsu.com \
--cc=mingo@elte.hu \
--cc=oleg@redhat.com \
--cc=rostedt@goodmis.org \
--cc=tzanussi@gmail.com \
--cc=zhaolei@cn.fujitsu.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.