From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933984AbbDJQGc (ORCPT ); Fri, 10 Apr 2015 12:06:32 -0400 Received: from mga09.intel.com ([134.134.136.24]:17574 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933624AbbDJQGX (ORCPT ); Fri, 10 Apr 2015 12:06:23 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.11,557,1422950400"; d="scan'208";a="711905447" From: Tom Zanussi To: rostedt@goodmis.org Cc: masami.hiramatsu.pt@hitachi.com, namhyung@kernel.org, andi@firstfloor.org, alexei.starovoitov@gmail.com, linux-kernel@vger.kernel.org, Tom Zanussi Subject: [PATCH v4 0/7] tracing: 'hist' triggers Date: Fri, 10 Apr 2015 11:05:51 -0500 Message-Id: X-Mailer: git-send-email 1.9.3 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This is v4 of the 'hist triggers' patchset, following feedback from v3. This version fixes the race in tracing_map_insert() noted in v3, where map.val.key could be checked even if map.val wasn't yet set. The simple fix for that in tracing_map_insert() introduces the possibility of duplicates in the map, which though rare, need to be accounted for in the output. To address that, duplicate-merging code was added to the map-printing code. It was also pointed out that it didn't seem correct to include module.h, but the fix for that has deeper roots and is being addressed by a separate patchset; for now we need to continue including module.h, though prompted by that I did some other header include cleanup. The functionality remains the same as v2, but this version no longer tries to export and use bpf_maps, and more importantly removes the associated GFP_NOTRACE/trace event hacks and kmem macros required to work around the bpf_map implementation. The tracing_map functionality is instead built on top of a simple lock-free map algorithm originated by Dr. Cliff Click (see references in the code for more details), which though too restrictive to be general-purpose in its current form, functions nicely as a special-purpose tracing map. v3 also moves the hist triggers code into a separate file and puts it all behind a new config option, CONFIG_HIST_TRIGGERS. It also merges in the sorting code rather than keeping it as a separate patch. This patchset also includes a couple other new and related triggers, enable_hist and disable_hist, very similar to the existing enable_event/disable_event triggers used to automatically enable and disable events based on a triggering condition, but in this case allowing hist triggers to be enabled and disabled in the same way. There are a couple of important bits of functionality that were present in v1 but not yet reimplemented in v3. The first is support for compound keys. Currently, maps can only be keyed on a single event field, whereas in v1 they could be keyed on multiple keys. With support for compound keys, you can create much more interesting output, such as for example per-pid lists of syscalls or read counts e.g.: # echo 'hist:keys=common_pid.execname,id.syscall:vals=hitcount' > \ /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger # cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist key: common_pid:bash[3112], id:sys_write vals: count:69 key: common_pid:bash[3112], id:sys_rt_sigprocmask vals: count:218 key: common_pid:update-notifier[3164], id:sys_poll vals: count:37 key: common_pid:update-notifier[3164], id:sys_recvfrom vals: count:118 key: common_pid:deja-dup-monito[3194], id:sys_sendto vals: count:1 key: common_pid:deja-dup-monito[3194], id:sys_read vals: count:4 key: common_pid:deja-dup-monito[3194], id:sys_poll vals: count:8 key: common_pid:deja-dup-monito[3194], id:sys_recvmsg vals: count:8 key: common_pid:deja-dup-monito[3194], id:sys_getegid vals: count:8 key: common_pid:emacs[3275], id:sys_fsync vals: count:1 key: common_pid:emacs[3275], id:sys_open vals: count:1 key: common_pid:emacs[3275], id:sys_symlink vals: count:2 key: common_pid:emacs[3275], id:sys_poll vals: count:23 key: common_pid:emacs[3275], id:sys_select vals: count:23 key: common_pid:emacs[3275], id:unknown_syscall vals: count:34 key: common_pid:emacs[3275], id:sys_ioctl vals: count:60 key: common_pid:emacs[3275], id:sys_rt_sigprocmask vals: count:116 key: common_pid:cat[3323], id:sys_munmap vals: count:1 key: common_pid:cat[3323], id:sys_fadvise64 vals: count:1 Related to that is support for sorting on multiple fields. Currently, you can sort using only a primary key. Being able to sort on multiple or at least a secondary key is indispensible for seeing trends when displaying multiple values. Changes from v3: - Added an insert check for val before checking the key associated with val - Added code to merge possible duplicates in the map Changes from v2: - reimplemented tracing_map, replacing bpf_map with nmi-safe/lock-free map - removed GPF_NOTRACE, kmalloc/free macros and event hacks needed by bpf_maps - moved hist triggers from trace_events_trigger.c to trace_events_hist.c - added CONFIG_HIST_TRIGGERS config option - consolidated sorting code with main patch Changes from v1: - completely rewritten on top of tracing_map (renamed and exported bpf_map) - added map clearing and client ops to tracing_map - changed the name from 'hash' triggers to 'hist' triggers - added new trigger 'pause' feature - added new enable_hist and disable_hist triggers - added usage for hist/enable_hist/disable hist to tracing/README - moved examples into Documentation/trace/event.txt - added ___GFP_NOTRACE, kmalloc/kfree macros, and conditional kmem tracepoints The following changes since commit e65e0516fb5fdfe1c3138ccd333651739894197f: Merge branch 'for-next/ftrace/core' into trace/for-next (2015-03-31 10:06:28 -0400) are available in the git repository at: git://git.yoctoproject.org/linux-yocto-contrib.git tzanussi/hist-triggers-v4 http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-contrib/log/?h=tzanussi/hist-triggers-v4 Tom Zanussi (7): tracing: Make ftrace_event_field checking functions available tracing: Add event record param to trigger_ops.func() tracing: Add get_syscall_name() tracing: Add a per-event-trigger 'paused' field tracing: Add 'hist' event trigger command tracing: Add enable_hist/disable_hist triggers tracing: Add 'hist' trigger Documentation Documentation/trace/events.txt | 870 ++++++++++++++++++ include/linux/ftrace_event.h | 9 +- kernel/trace/Kconfig | 14 + kernel/trace/Makefile | 1 + kernel/trace/trace.c | 54 ++ kernel/trace/trace.h | 77 +- kernel/trace/trace_events.c | 4 + kernel/trace/trace_events_filter.c | 12 - kernel/trace/trace_events_hist.c | 1708 +++++++++++++++++++++++++++++++++++ kernel/trace/trace_events_trigger.c | 139 +-- kernel/trace/trace_syscalls.c | 11 + 11 files changed, 2819 insertions(+), 80 deletions(-) create mode 100644 kernel/trace/trace_events_hist.c -- 1.9.3