linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 00/34] perf clang: Builtin clang and perfhook support
@ 2016-11-15  4:05 Wang Nan
  2016-11-15  4:05 ` [PATCH 01/34] perf tools: Fix kernel version error in ubuntu Wang Nan
                   ` (34 more replies)
  0 siblings, 35 replies; 46+ messages in thread
From: Wang Nan @ 2016-11-15  4:05 UTC (permalink / raw)
  To: acme, ast; +Cc: lizefan, hekuang, linux-kernel, pi3orama, Wang Nan, Jiri Olsa

This is version 2 of perf builtin clang patch series. Compare to v1,
add an exciting feature: jit compiling perf hook functions. This
features allows script writer report result through BPF map in a
customized way.

At the end of this cover letter lists an example shows how to capture
and report a syscall histogram.

This patchset is based on current perf/core.

In this patchset:

 Patch  1 -  4 are bugfixes left in my local tree.
 Patch  5 -  7 are preparation in libbpf.
 Patch  8 -  9 introduce perf hook to perf.
 Patch 10 - 21 are patches support builtin clang. Some of then are already
               collected by Arnaldo in tmp.perf/builtin-clang. They are slightly
	       adjusted in this v2 series.
 Patch 22 - 29 add JIT compiling to builting clang and add them to perf hook.
 Patch 30 - 34 are easy of use improvements. 1) builtin clang defines macros
               and helpers by default using builtin include headers. Default
	       headers can be turned off by -UBUILTIN_CLANG_DEFAULT_INCLUDE.
	       2) allow JITted perf hooks access more functions in libc.

Example: build syscall histogram, exclude syscalls issued by perf itself.
         Please note following improvements:
	 1. Don't need define many bpf helpers by hand.
	    See bpf_map_lookup_elem and bpf_get_current_pid_tgid.
	 2. See how pid of git is passed to BPF script attached to
	    raw_syscalls:sys_enter.

  $ cat ./count_syscalls.c
  typedef unsigned long u64;
  
  #define BPF_MAP_TYPE_HASH 1
  #define BPF_MAP_TYPE_ARRAY 2
  
  enum GVAL {
  	G_perf_pid,
  	NR_GVALS
  };
  
  struct bpf_map_def SEC("maps") GVALS = {
  	.type = BPF_MAP_TYPE_ARRAY,
  	.key_size = sizeof(int),
  	.value_size = sizeof(u64),
  	.max_entries = NR_GVALS,
  };
  
  struct bpf_map_def SEC("maps") syscall_counter = {
  	.type = BPF_MAP_TYPE_HASH,
  	.key_size = sizeof(u64),
  	.value_size = sizeof(u64),
  	.max_entries = 512,
  };
  
  SEC("raw_syscalls:sys_enter")
  int func(void *ctx)
  {
  	int key = G_perf_pid;
  	u64 id = *((u64 *)(ctx + 8));
  	int self_pid = bpf_get_current_pid_tgid() & 0xffffffff;
  	int *perf_pid = bpf_map_lookup_elem(&GVALS, &key);
  	u64 *counter;
  
  	if (!perf_pid)
  		return 0;
  	if (*perf_pid == self_pid)
  		return 0;
  	counter = bpf_map_lookup_elem(&syscall_counter, &id);
  	if (!counter) {
  		u64 value = 1;
  		bpf_map_update_elem(&syscall_counter, &id, &value, 0);
  		return 0;
  	}
  	__sync_fetch_and_add(counter, 1);
  	return 0;
  }
  
  SEC("perfhook:record_start")
  void record_start(void *ctx)
  {
  	int perf_pid = getpid(), key = G_perf_pid;
  	printf("Start count, perfpid=%d\n", perf_pid);
  	jit_helper__map_update_elem(ctx, &GVALS, &key, &perf_pid, 0);
  }
  
  SEC("perfhook:record_end")
  void record_end(void *ctx)
  {
  	u64 key = -1, value;
  	while (!jit_helper__map_get_next_key(ctx, &syscall_counter, &key, &key)) {
  		jit_helper__map_lookup_elem(ctx, &syscall_counter, &key, &value);
  		printf("syscall %ld\tcount: %ld\n", (long)key, (long)value);
  	}
  }
  char _license[] SEC("license") = "GPL";
  int _version SEC("version") = LINUX_VERSION_CODE;

  $ sudo -s
  # ulimit -l unlimited
  # perf record -e ./count_syscalls.c echo "Haha"
  Start count, perfpid=25209
  Haha
  [ perf record: Woken up 1 times to write data ]
  syscall 8	count: 6
  syscall 11	count: 1
  syscall 4	count: 6
  syscall 21	count: 1
  syscall 5	count: 3
  syscall 231	count: 1
  syscall 45	count: 3
  syscall 0	count: 24
  syscall 257	count: 1
  syscall 59	count: 4
  syscall 23	count: 9
  syscall 78	count: 2
  syscall 41	count: 4
  syscall 72	count: 8
  syscall 10	count: 3
  syscall 321	count: 1
  syscall 298	count: 7
  syscall 16	count: 21
  syscall 9	count: 16
  syscall 1	count: 114
  syscall 12	count: 3
  syscall 14	count: 35
  syscall 158	count: 1
  syscall 2	count: 15
  syscall 7	count: 18
  syscall 3	count: 11
  [ perf record: Captured and wrote 0.011 MB perf.data ]

Eric Leblond (1):
  tools lib bpf: fix maps resolution

Wang Nan (33):
  perf tools: Fix kernel version error in ubuntu
  perf record: Fix segfault when running with suid and kptr_restrict is
    1
  tools perf: Add missing struct defeinition in probe_event.h
  tools lib bpf: Add missing bpf map functions
  tools lib bpf: Add private field for bpf_object
  tools lib bpf: Retrive bpf_map through offset of bpf_map_def
  perf tools: Introduce perf hooks
  perf tools: Pass context to perf hook functions
  perf llvm: Extract helpers in llvm-utils.c
  tools build: Add feature detection for LLVM
  tools build: Add feature detection for clang
  perf build: Add clang and llvm compile and linking support
  perf clang: Add builtin clang support ant test case
  perf clang: Use real file system for #include
  perf clang: Allow passing CFLAGS to builtin clang
  perf clang: Update test case to use real BPF script
  perf clang: Support compile IR to BPF object and add testcase
  perf clang: Compile BPF script use builtin clang support
  perf clang: Pass full path to builtin clang
  perf clang: Pass CFLAGS to builtin clang
  perf clang jit: Wrap llvm::Module using PerfModule
  perf clang jit: Insignt BPF and JIT functions in a Module
  perf clang jit: add PerfModule::doJIT to JIT perfhook functions.
  perf clang jit: Export functions for jitted code
  perf clang jit: Actually JIT and hook in bpf loader
  perf clang jit: Collect the lowest address in maps section as map_base
  perf clang jit: Access BPF map
  perf clang jit: Allow jitted perf hook access BPF maps
  perf clang: Link BPF functions declaration into perf
  perf clang: Declare BPF functions for BPF scripts automatically
  perf clang: Include helpers to BPF scripts
  perf clang builtin: Define hook helpers by default
  perf clang git: Export getpid() to perf hook

 tools/build/feature/Makefile                  |  18 +
 tools/build/feature/test-clang.cpp            |  21 ++
 tools/build/feature/test-llvm.cpp             |   8 +
 tools/lib/bpf/bpf.c                           |  56 +++
 tools/lib/bpf/bpf.h                           |   7 +
 tools/lib/bpf/libbpf.c                        | 177 ++++++---
 tools/lib/bpf/libbpf.h                        |  13 +
 tools/perf/Makefile.config                    |  62 +++-
 tools/perf/Makefile.perf                      |  23 +-
 tools/perf/builtin-record.c                   |  11 +
 tools/perf/tests/Build                        |   4 +-
 tools/perf/tests/bpf-script-example.c         |  30 +-
 tools/perf/tests/bpf-script-test-kbuild.c     |   2 +
 tools/perf/tests/bpf-script-test-prologue.c   |   6 +-
 tools/perf/tests/bpf-script-test-relocation.c |  17 +-
 tools/perf/tests/builtin-test.c               |  13 +
 tools/perf/tests/clang.c                      |  50 +++
 tools/perf/tests/llvm.h                       |   7 +
 tools/perf/tests/make                         |   2 +
 tools/perf/tests/perf-hooks.c                 |  48 +++
 tools/perf/tests/tests.h                      |   4 +
 tools/perf/util/Build                         |   5 +
 tools/perf/util/bpf-loader.c                  | 102 +++++-
 tools/perf/util/bpf-loader.h                  |  19 +
 tools/perf/util/c++/Build                     |   4 +
 tools/perf/util/c++/bpf-funcs-str.c           | 228 ++++++++++++
 tools/perf/util/c++/bpf-helper-str.c          |  24 ++
 tools/perf/util/c++/clang-bpf-includes.h      |  13 +
 tools/perf/util/c++/clang-c.h                 |  63 ++++
 tools/perf/util/c++/clang-test.cpp            |  99 +++++
 tools/perf/util/c++/clang.cpp                 | 498 ++++++++++++++++++++++++++
 tools/perf/util/c++/clang.h                   |  61 ++++
 tools/perf/util/jit-helpers.c                 |  60 ++++
 tools/perf/util/jit-helpers.h                 |  30 ++
 tools/perf/util/llvm-utils.c                  |  76 +++-
 tools/perf/util/llvm-utils.h                  |  15 +-
 tools/perf/util/perf-hooks-list.h             |   3 +
 tools/perf/util/perf-hooks.c                  |  88 +++++
 tools/perf/util/perf-hooks.h                  |  39 ++
 tools/perf/util/probe-event.h                 |   2 +
 tools/perf/util/symbol.c                      |   2 +-
 tools/perf/util/util-cxx.h                    |  26 ++
 tools/perf/util/util.c                        |  55 ++-
 43 files changed, 1991 insertions(+), 100 deletions(-)
 create mode 100644 tools/build/feature/test-clang.cpp
 create mode 100644 tools/build/feature/test-llvm.cpp
 create mode 100644 tools/perf/tests/clang.c
 create mode 100644 tools/perf/tests/perf-hooks.c
 create mode 100644 tools/perf/util/c++/Build
 create mode 100644 tools/perf/util/c++/bpf-funcs-str.c
 create mode 100644 tools/perf/util/c++/bpf-helper-str.c
 create mode 100644 tools/perf/util/c++/clang-bpf-includes.h
 create mode 100644 tools/perf/util/c++/clang-c.h
 create mode 100644 tools/perf/util/c++/clang-test.cpp
 create mode 100644 tools/perf/util/c++/clang.cpp
 create mode 100644 tools/perf/util/c++/clang.h
 create mode 100644 tools/perf/util/jit-helpers.c
 create mode 100644 tools/perf/util/jit-helpers.h
 create mode 100644 tools/perf/util/perf-hooks-list.h
 create mode 100644 tools/perf/util/perf-hooks.c
 create mode 100644 tools/perf/util/perf-hooks.h
 create mode 100644 tools/perf/util/util-cxx.h

Signed-off-by: Wang Nan <wangnan0@huawei.com>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com

-- 
2.10.1

^ permalink raw reply	[flat|nested] 46+ messages in thread
* Re: [PATCH 00/34] perf clang: Builtin clang and perfhook support
@ 2016-11-15  4:57 Alexei Starovoitov
  2016-11-15  5:03 ` Wangnan (F)
  0 siblings, 1 reply; 46+ messages in thread
From: Alexei Starovoitov @ 2016-11-15  4:57 UTC (permalink / raw)
  To: Wang Nan
  Cc: Arnaldo Carvalho de Melo, Alexei Starovoitov, Zefan Li, He Kuang,
	linux-kernel, pi3orama, Jiri Olsa

On Mon, Nov 14, 2016 at 8:05 PM, Wang Nan <wangnan0@huawei.com> wrote:
> This is version 2 of perf builtin clang patch series. Compare to v1,
> add an exciting feature: jit compiling perf hook functions. This
> features allows script writer report result through BPF map in a
> customized way.

looks great.

>   SEC("perfhook:record_start")
>   void record_start(void *ctx)
>   {
>         int perf_pid = getpid(), key = G_perf_pid;
>         printf("Start count, perfpid=%d\n", perf_pid);
>         jit_helper__map_update_elem(ctx, &GVALS, &key, &perf_pid, 0);

the name, I think, is too verbose.
Why not to keep them as bpf_map_update_elem
even for user space programs?

>   SEC("perfhook:record_end")
>   void record_end(void *ctx)
>   {
>         u64 key = -1, value;
>         while (!jit_helper__map_get_next_key(ctx, &syscall_counter, &key, &key)) {
>                 jit_helper__map_lookup_elem(ctx, &syscall_counter, &key, &value);
>                 printf("syscall %ld\tcount: %ld\n", (long)key, (long)value);

this loop will be less verbose as well.

^ permalink raw reply	[flat|nested] 46+ messages in thread
* Re: [PATCH 00/34] perf clang: Builtin clang and perfhook support
@ 2016-11-15  5:21 Alexei Starovoitov
  2016-11-15  5:37 ` Wangnan (F)
  0 siblings, 1 reply; 46+ messages in thread
From: Alexei Starovoitov @ 2016-11-15  5:21 UTC (permalink / raw)
  To: Wangnan (F)
  Cc: Arnaldo Carvalho de Melo, Alexei Starovoitov, Zefan Li, He Kuang,
	linux-kernel, pi3orama, Jiri Olsa

On Mon, Nov 14, 2016 at 9:03 PM, Wangnan (F) <wangnan0@huawei.com> wrote:
>
>
> On 2016/11/15 12:57, Alexei Starovoitov wrote:
>>
>> On Mon, Nov 14, 2016 at 8:05 PM, Wang Nan <wangnan0@huawei.com> wrote:
>>>
>>> This is version 2 of perf builtin clang patch series. Compare to v1,
>>> add an exciting feature: jit compiling perf hook functions. This
>>> features allows script writer report result through BPF map in a
>>> customized way.
>>
>> looks great.
>>
>>>    SEC("perfhook:record_start")
>>>    void record_start(void *ctx)
>>>    {
>>>          int perf_pid = getpid(), key = G_perf_pid;
>>>          printf("Start count, perfpid=%d\n", perf_pid);
>>>          jit_helper__map_update_elem(ctx, &GVALS, &key, &perf_pid, 0);
>>
>> the name, I think, is too verbose.
>> Why not to keep them as bpf_map_update_elem
>> even for user space programs?
>
>
> I can make it shorter by give it a better name or use a wrapper like
>
> BPF_MAP(update_elem)

the macro isn't pretty, since function calls won't look like calls.

> but the only thing I can't do is to make perfhook and in-kernel script
> use a uniform name for these bpf_map functions, because
> bpf_map_update_elem is already defined:
>
> "static long (*bpf_map_update_elem)(void *, void *, void *, unsigned long) =
> (void *)2;\n"

right. i guess you could have #ifdef it, so it's different for bpf backend
and for native.
Another alternative is to call it map_update_elem or map_update
or bpf_map_update. Something shorter is already a win.
'jit_helper__' prefix is an implementation detail. The users don't
need to know and don't need to spell it out everywhere.

^ permalink raw reply	[flat|nested] 46+ messages in thread

end of thread, other threads:[~2016-11-25 17:23 UTC | newest]

Thread overview: 46+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-15  4:05 [PATCH 00/34] perf clang: Builtin clang and perfhook support Wang Nan
2016-11-15  4:05 ` [PATCH 01/34] perf tools: Fix kernel version error in ubuntu Wang Nan
2016-11-25 17:20   ` [tip:perf/core] " tip-bot for Wang Nan
2016-11-15  4:05 ` [PATCH 02/34] perf record: Fix segfault when running with suid and kptr_restrict is 1 Wang Nan
2016-11-25 17:21   ` [tip:perf/core] " tip-bot for Wang Nan
2016-11-15  4:05 ` [PATCH 03/34] tools perf: Add missing struct defeinition in probe_event.h Wang Nan
2016-11-25 17:21   ` [tip:perf/core] perf tools: Add missing struct definition " tip-bot for Wang Nan
2016-11-15  4:05 ` [PATCH 04/34] tools lib bpf: fix maps resolution Wang Nan
2016-11-25 17:22   ` [tip:perf/core] tools lib bpf: Fix " tip-bot for Eric Leblond
2016-11-15  4:05 ` [PATCH 05/34] tools lib bpf: Add missing bpf map functions Wang Nan
2016-11-17  3:23   ` Wangnan (F)
2016-11-25 14:31     ` Arnaldo Carvalho de Melo
2016-11-15  4:05 ` [PATCH 06/34] tools lib bpf: Add private field for bpf_object Wang Nan
2016-11-15  4:05 ` [PATCH 07/34] tools lib bpf: Retrive bpf_map through offset of bpf_map_def Wang Nan
2016-11-15  4:05 ` [PATCH 08/34] perf tools: Introduce perf hooks Wang Nan
2016-11-15  4:05 ` [PATCH 09/34] perf tools: Pass context to perf hook functions Wang Nan
2016-11-15  4:05 ` [PATCH 10/34] perf llvm: Extract helpers in llvm-utils.c Wang Nan
2016-11-15  4:05 ` [PATCH 11/34] tools build: Add feature detection for LLVM Wang Nan
2016-11-15  4:05 ` [PATCH 12/34] tools build: Add feature detection for clang Wang Nan
2016-11-15  4:05 ` [PATCH 13/34] perf build: Add clang and llvm compile and linking support Wang Nan
2016-11-15  4:05 ` [PATCH 14/34] perf clang: Add builtin clang support ant test case Wang Nan
2016-11-15  4:05 ` [PATCH 15/34] perf clang: Use real file system for #include Wang Nan
2016-11-15  4:05 ` [PATCH 16/34] perf clang: Allow passing CFLAGS to builtin clang Wang Nan
2016-11-15  4:06 ` [PATCH 17/34] perf clang: Update test case to use real BPF script Wang Nan
2016-11-15  4:06 ` [PATCH 18/34] perf clang: Support compile IR to BPF object and add testcase Wang Nan
2016-11-15  4:06 ` [PATCH 19/34] perf clang: Compile BPF script use builtin clang support Wang Nan
2016-11-15  4:06 ` [PATCH 20/34] perf clang: Pass full path to builtin clang Wang Nan
2016-11-15  4:06 ` [PATCH 21/34] perf clang: Pass CFLAGS " Wang Nan
2016-11-15  4:06 ` [PATCH 22/34] perf clang jit: Wrap llvm::Module using PerfModule Wang Nan
2016-11-15  4:06 ` [PATCH 23/34] perf clang jit: Insignt BPF and JIT functions in a Module Wang Nan
2016-11-15  4:06 ` [PATCH 24/34] perf clang jit: add PerfModule::doJIT to JIT perfhook functions Wang Nan
2016-11-15  4:06 ` [PATCH 25/34] perf clang jit: Export functions for jitted code Wang Nan
2016-11-15  4:06 ` [PATCH 26/34] perf clang jit: Actually JIT and hook in bpf loader Wang Nan
2016-11-15  4:06 ` [PATCH 27/34] perf clang jit: Collect the lowest address in maps section as map_base Wang Nan
2016-11-15  4:06 ` [PATCH 28/34] perf clang jit: Access BPF map Wang Nan
2016-11-15  4:06 ` [PATCH 29/34] perf clang jit: Allow jitted perf hook access BPF maps Wang Nan
2016-11-15  4:06 ` [PATCH 30/34] perf clang: Link BPF functions declaration into perf Wang Nan
2016-11-15  4:06 ` [PATCH 31/34] perf clang: Declare BPF functions for BPF scripts automatically Wang Nan
2016-11-15  4:06 ` [PATCH 32/34] perf clang: Include helpers to BPF scripts Wang Nan
2016-11-15  4:06 ` [PATCH 33/34] perf clang builtin: Define hook helpers by default Wang Nan
2016-11-15  4:06 ` [PATCH 34/34] perf clang git: Export getpid() to perf hook Wang Nan
2016-11-15  4:32 ` [PATCH 00/34] perf clang: Builtin clang and perfhook support Wangnan (F)
2016-11-15  4:57 Alexei Starovoitov
2016-11-15  5:03 ` Wangnan (F)
2016-11-15  5:21 Alexei Starovoitov
2016-11-15  5:37 ` Wangnan (F)

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).