linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCHv3 00/25] perf: Add mmap2 build id support
@ 2020-11-23 23:04 Jiri Olsa
  2020-11-23 23:04 ` [PATCH 01/25] bpf: Move stack_map_get_build_id into lib Jiri Olsa
                   ` (25 more replies)
  0 siblings, 26 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Frank Ch . Eigler, Mark Wielaard, lkml, Peter Zijlstra,
	Ingo Molnar, Mark Rutland, Namhyung Kim, Alexander Shishkin,
	Michael Petlan, Song Liu, Ian Rogers, Stephane Eranian,
	Alexey Budankov, Andi Kleen, Adrian Hunter

hi,
adding the support to have buildid stored in mmap2 event,
so we can bypass the final perf record hunt on build ids.

This patchset allows perf to record build ID in mmap2 event,
and adds perf tooling to store/download binaries to .debug
cache based on these build IDs.

Note that the build id retrieval code is stolen from bpf
code, where it's been used (together with file offsets)
to replace IPs in user space stack traces. It's now added
under lib directory.

v3 changes:
  - added acks
  - removed forgotten debug code [Arnaldo]
  - fixed readlink termination [Ian]
  - fixed doc for --debuginfod=URLs [Ian]
  - adopted kernel's memchr_inv function and used
    it in build_id__is_defined function [Arnaldo]

On recording server:

  - on the recording server we can run record with --buildid-mmap
    option to store build ids in mmap2 events:

    # perf record --buildid-mmap
    ^C[ perf record: Woken up 2 times to write data ]
    [ perf record: Captured and wrote 0.836 MB perf.data ]

  - it stores nothing to ~/.debug cache:

    # find ~/.debug
    find: ‘/root/.debug’: No such file or directory

  - and still reports properly:

    # perf report --stdio
    ...
    99.82%  swapper          [kernel.kallsyms]  [k] native_safe_halt
     0.03%  swapper          [kernel.kallsyms]  [k] finish_task_switch
     0.02%  swapper          [kernel.kallsyms]  [k] __softirqentry_text_start
     0.01%  kcompactd0       [kernel.kallsyms]  [k] _raw_spin_unlock_irqrestore
     0.01%  ksoftirqd/6      [kernel.kallsyms]  [k] slab_free_freelist_hook
     0.01%  kworker/17:1H-x  [kernel.kallsyms]  [k] slab_free_freelist_hook

  - display used/hit build ids:

    # perf buildid-list | head -5
    5dcec522abf136fcfd3128f47e131f2365834dd7 /proc/kcore
    589e403a34f55486bcac848a45e00bcdeedd1ca8 /usr/lib64/libcrypto.so.1.1.1g
    94569566d4eac7e9c87ba029d43d4e2158f9527e /usr/lib64/libpthread-2.30.so
    559b9702bebe31c6d132c8dc5cc887673d65d5b5 /usr/lib64/libc-2.30.so
    40da7abe89f631f60538a17686a7d65c6a02ed31 /usr/lib64/ld-2.30.so

  - store build id binaries into build id cache:

    # perf buildid-cache -a perf.data
    OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /proc/kcore
    OK   589e403a34f55486bcac848a45e00bcdeedd1ca8 /usr/lib64/libcrypto.so.1.1.1g
    OK   94569566d4eac7e9c87ba029d43d4e2158f9527e /usr/lib64/libpthread-2.30.so
    OK   559b9702bebe31c6d132c8dc5cc887673d65d5b5 /usr/lib64/libc-2.30.so
    OK   40da7abe89f631f60538a17686a7d65c6a02ed31 /usr/lib64/ld-2.30.so
    OK   a674f7a47c78e35a088104647b9640710277b489 /usr/sbin/sshd
    OK   e5cb4ca25f46485bdbc691c3a92e7e111dac3ef2 /usr/bin/bash
    OK   9bc8589108223c944b452f0819298a0c3cba6215 /usr/bin/find

    # find ~/.debug | head -5
    /root/.debug
    /root/.debug/proc
    /root/.debug/proc/kcore
    /root/.debug/proc/kcore/5dcec522abf136fcfd3128f47e131f2365834dd7
    /root/.debug/proc/kcore/5dcec522abf136fcfd3128f47e131f2365834dd7/kallsyms

  - run debuginfod daemon to provide binaries to another server (below)
    (the initialization could take some time)

    # debuginfod -F /


On another server:

  - copy perf.data from 'record' server and run:

    $ find ~/.debug/
    find: ‘/home/jolsa/.debug/’: No such file or directory

    $ perf buildid-list | head -5
    No kallsyms or vmlinux with build-id 5dcec522abf136fcfd3128f47e131f2365834dd7 was found
    5dcec522abf136fcfd3128f47e131f2365834dd7 [kernel.kallsyms]
    5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
    589e403a34f55486bcac848a45e00bcdeedd1ca8 /usr/lib64/libcrypto.so.1.1.1g
    94569566d4eac7e9c87ba029d43d4e2158f9527e /usr/lib64/libpthread-2.30.so
    559b9702bebe31c6d132c8dc5cc887673d65d5b5 /usr/lib64/libc-2.30.so

  - report does not show anything (kernel build id does not match):

   $ perf report --stdio
   ...
    76.73%  swapper          [kernel.kallsyms]    [k] 0xffffffff81aa8ebe
     1.89%  find             [kernel.kallsyms]    [k] 0xffffffff810f2167
     0.93%  sshd             [kernel.kallsyms]    [k] 0xffffffff8153380c
     0.83%  swapper          [kernel.kallsyms]    [k] 0xffffffff81104b0b
     0.71%  kworker/u40:2-e  [kernel.kallsyms]    [k] 0xffffffff810f3850
     0.70%  kworker/u40:0-e  [kernel.kallsyms]    [k] 0xffffffff810f3850
     0.64%  find             [kernel.kallsyms]    [k] 0xffffffff81a9ba0a
     0.63%  find             [kernel.kallsyms]    [k] 0xffffffff81aa93b0

  - add build ids does not work, because existing binaries (on another server)
    have different build ids:

    $ perf buildid-cache -a perf.data 
    No kallsyms or vmlinux with build-id 5dcec522abf136fcfd3128f47e131f2365834dd7 was found
    FAIL 5dcec522abf136fcfd3128f47e131f2365834dd7 [kernel.kallsyms]
    FAIL 5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
    FAIL 589e403a34f55486bcac848a45e00bcdeedd1ca8 /usr/lib64/libcrypto.so.1.1.1g
    FAIL 94569566d4eac7e9c87ba029d43d4e2158f9527e /usr/lib64/libpthread-2.30.so
    FAIL 559b9702bebe31c6d132c8dc5cc887673d65d5b5 /usr/lib64/libc-2.30.so
    FAIL 40da7abe89f631f60538a17686a7d65c6a02ed31 /usr/lib64/ld-2.30.so
    FAIL a674f7a47c78e35a088104647b9640710277b489 /usr/sbin/sshd
    FAIL e5cb4ca25f46485bdbc691c3a92e7e111dac3ef2 /usr/bin/bash
    FAIL 9bc8589108223c944b452f0819298a0c3cba6215 /usr/bin/find

  - add build ids with debuginfod setup pointing to record server:

    $ perf buildid-cache -a perf.data --debuginfod http://192.168.122.174:8002
    No kallsyms or vmlinux with build-id 5dcec522abf136fcfd3128f47e131f2365834dd7 was found
    OK   5dcec522abf136fcfd3128f47e131f2365834dd7 [kernel.kallsyms]
    OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
    OK   589e403a34f55486bcac848a45e00bcdeedd1ca8 /usr/lib64/libcrypto.so.1.1.1g
    OK   94569566d4eac7e9c87ba029d43d4e2158f9527e /usr/lib64/libpthread-2.30.so
    OK   559b9702bebe31c6d132c8dc5cc887673d65d5b5 /usr/lib64/libc-2.30.so
    OK   40da7abe89f631f60538a17686a7d65c6a02ed31 /usr/lib64/ld-2.30.so
    OK   a674f7a47c78e35a088104647b9640710277b489 /usr/sbin/sshd
    OK   e5cb4ca25f46485bdbc691c3a92e7e111dac3ef2 /usr/bin/bash
    OK   9bc8589108223c944b452f0819298a0c3cba6215 /usr/bin/find

  - and report works:

    $ perf report --stdio
    ...
    76.73%  swapper          [kernel.kallsyms]    [k] native_safe_halt
     1.91%  find             [kernel.kallsyms]    [k] queue_work_on
     0.93%  sshd             [kernel.kallsyms]    [k] iowrite16
     0.83%  swapper          [kernel.kallsyms]    [k] finish_task_switch
     0.72%  kworker/u40:2-e  [kernel.kallsyms]    [k] process_one_work
     0.70%  kworker/u40:0-e  [kernel.kallsyms]    [k] process_one_work
     0.64%  find             [kernel.kallsyms]    [k] syscall_enter_from_user_mode
     0.63%  find             [kernel.kallsyms]    [k] _raw_spin_unlock_irqrestore

  - because we have the data in build id cache:

    $ find ~/.debug | head -10
    .../.debug
    .../.debug/home
    .../.debug/home/jolsa
    .../.debug/home/jolsa/.cache
    .../.debug/home/jolsa/.cache/debuginfod_client
    .../.debug/home/jolsa/.cache/debuginfod_client/5dcec522abf136fcfd3128f47e131f2365834dd7
    .../.debug/home/jolsa/.cache/debuginfod_client/5dcec522abf136fcfd3128f47e131f2365834dd7/executable
    .../.debug/home/jolsa/.cache/debuginfod_client/5dcec522abf136fcfd3128f47e131f2365834dd7/executable/5dcec522abf136fcfd3128f47e131f2365834dd7
    .../.debug/home/jolsa/.cache/debuginfod_client/5dcec522abf136fcfd3128f47e131f2365834dd7/executable/5dcec522abf136fcfd3128f47e131f2365834dd7/elf
    .../.debug/home/jolsa/.cache/debuginfod_client/5dcec522abf136fcfd3128f47e131f2365834dd7/executable/5dcec522abf136fcfd3128f47e131f2365834dd7/debug


Available also in:
  git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
  perf/build_id

thanks,
jirka


Cc: Frank Ch. Eigler <fche@redhat.com>
Cc: Mark Wielaard <mjw@redhat.com>
---
Jiri Olsa (25):
      bpf: Move stack_map_get_build_id into lib
      bpf: Add size arg to build_id_parse function
      perf: Add build id data in mmap2 event
      tools headers uapi: Sync tools/include/uapi/linux/perf_event.h
      tools lib: Adopt memchr_inv() from kernel
      perf tools: Do not swap mmap2 fields in case it contains build id
      perf tools: Add build_id__is_defined function
      perf tools: Add filename__decompress function
      perf tools: Add support to read build id from compressed elf
      perf tools: Add check for existing link in buildid dir
      perf tools: Use struct extra_kernel_map in machine__process_kernel_mmap_event
      perf tools: Try to load vmlinux from buildid database
      perf tools: Store build id from mmap2 events
      perf tools: Allow mmap2 event to synthesize kernel image
      perf tools: Allow mmap2 event to synthesize modules
      perf tools: Synthesize build id for kernel/modules/tasks
      perf tools: Add support to display build id for mmap2 events
      perf tools: Use machine__for_each_dso in perf_session__cache_build_ids
      perf tools: Add __perf_session__cache_build_ids function
      perf tools: Add is_perf_data function
      perf tools: Add build_id_cache__add function
      perf buildid-cache: Add support to add build ids from perf data
      perf buildid-cache: Add --debuginfod option
      perf buildid-list: Add support for mmap2's buildid events
      perf record: Add --buildid-mmap option to enable mmap's build id

 include/linux/buildid.h                           |  12 +++++
 include/uapi/linux/perf_event.h                   |  42 +++++++++++++++---
 kernel/bpf/stackmap.c                             | 143 ++---------------------------------------------------------
 kernel/events/core.c                              |  32 ++++++++++++--
 lib/Makefile                                      |   3 +-
 lib/buildid.c                                     | 149 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/include/linux/string.h                      |   1 +
 tools/include/uapi/linux/perf_event.h             |  42 +++++++++++++++---
 tools/lib/perf/include/perf/event.h               |  18 ++++++--
 tools/lib/string.c                                |  58 ++++++++++++++++++++++++
 tools/perf/Documentation/perf-buildid-cache.txt   |  18 +++++++-
 tools/perf/Documentation/perf-config.txt          |  10 ++++-
 tools/perf/Documentation/perf-record.txt          |   3 ++
 tools/perf/builtin-buildid-cache.c                | 243 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 tools/perf/builtin-buildid-list.c                 |   3 ++
 tools/perf/builtin-record.c                       |  20 +++++++++
 tools/perf/tests/shell/trace+probe_vfs_getname.sh |   2 +-
 tools/perf/util/build-id.c                        | 127 ++++++++++++++++++++++++++++++++++-------------------
 tools/perf/util/build-id.h                        |   8 ++++
 tools/perf/util/data.c                            |  19 ++++++++
 tools/perf/util/data.h                            |   1 +
 tools/perf/util/dso.c                             |  31 ++++++++-----
 tools/perf/util/dso.h                             |   2 +
 tools/perf/util/event.c                           |  41 ++++++++++++-----
 tools/perf/util/evsel.c                           |  10 +++--
 tools/perf/util/machine.c                         |  80 ++++++++++++++++++++-------------
 tools/perf/util/map.c                             |   8 +++-
 tools/perf/util/map.h                             |   3 +-
 tools/perf/util/perf_api_probe.c                  |  10 +++++
 tools/perf/util/perf_api_probe.h                  |   1 +
 tools/perf/util/perf_event_attr_fprintf.c         |   2 +
 tools/perf/util/probe-event.c                     |   6 +--
 tools/perf/util/record.h                          |   1 +
 tools/perf/util/session.c                         |  11 +++--
 tools/perf/util/symbol-elf.c                      |  37 +++++++++++++++-
 tools/perf/util/symbol.c                          |  16 +++++++
 tools/perf/util/symbol_conf.h                     |   3 +-
 tools/perf/util/synthetic-events.c                | 121 +++++++++++++++++++++++++++++++++++++-------------
 38 files changed, 1026 insertions(+), 311 deletions(-)
 create mode 100644 include/linux/buildid.h
 create mode 100644 lib/buildid.c


^ permalink raw reply	[flat|nested] 37+ messages in thread

* [PATCH 01/25] bpf: Move stack_map_get_build_id into lib
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 02/25] bpf: Add size arg to build_id_parse function Jiri Olsa
                   ` (24 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Alexei Starovoitov, Song Liu, lkml, Peter Zijlstra, Ingo Molnar,
	Mark Rutland, Namhyung Kim, Alexander Shishkin, Michael Petlan,
	Ian Rogers, Stephane Eranian, Alexey Budankov, Andi Kleen,
	Adrian Hunter

Moving stack_map_get_build_id into lib with
declaration in linux/buildid.h header:

  int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id);

This function returns build id for given struct vm_area_struct.
There is no functional change to stack_map_get_build_id function.

Cc: Alexei Starovoitov <ast@kernel.org>
Acked-by: Song Liu <songliubraving@fb.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 include/linux/buildid.h |  11 ++++
 kernel/bpf/stackmap.c   | 143 ++--------------------------------------
 lib/Makefile            |   3 +-
 lib/buildid.c           | 136 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 153 insertions(+), 140 deletions(-)
 create mode 100644 include/linux/buildid.h
 create mode 100644 lib/buildid.c

diff --git a/include/linux/buildid.h b/include/linux/buildid.h
new file mode 100644
index 000000000000..3be5b49719f1
--- /dev/null
+++ b/include/linux/buildid.h
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_BUILDID_H
+#define _LINUX_BUILDID_H
+
+#include <linux/mm_types.h>
+
+#define BUILD_ID_SIZE 20
+
+int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id);
+
+#endif
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
index 06065fa27124..7df08f8af5a1 100644
--- a/kernel/bpf/stackmap.c
+++ b/kernel/bpf/stackmap.c
@@ -7,10 +7,9 @@
 #include <linux/kernel.h>
 #include <linux/stacktrace.h>
 #include <linux/perf_event.h>
-#include <linux/elf.h>
-#include <linux/pagemap.h>
 #include <linux/irq_work.h>
 #include <linux/btf_ids.h>
+#include <linux/buildid.h>
 #include "percpu_freelist.h"
 
 #define STACK_CREATE_FLAG_MASK					\
@@ -153,140 +152,6 @@ static struct bpf_map *stack_map_alloc(union bpf_attr *attr)
 	return ERR_PTR(err);
 }
 
-#define BPF_BUILD_ID 3
-/*
- * Parse build id from the note segment. This logic can be shared between
- * 32-bit and 64-bit system, because Elf32_Nhdr and Elf64_Nhdr are
- * identical.
- */
-static inline int stack_map_parse_build_id(void *page_addr,
-					   unsigned char *build_id,
-					   void *note_start,
-					   Elf32_Word note_size)
-{
-	Elf32_Word note_offs = 0, new_offs;
-
-	/* check for overflow */
-	if (note_start < page_addr || note_start + note_size < note_start)
-		return -EINVAL;
-
-	/* only supports note that fits in the first page */
-	if (note_start + note_size > page_addr + PAGE_SIZE)
-		return -EINVAL;
-
-	while (note_offs + sizeof(Elf32_Nhdr) < note_size) {
-		Elf32_Nhdr *nhdr = (Elf32_Nhdr *)(note_start + note_offs);
-
-		if (nhdr->n_type == BPF_BUILD_ID &&
-		    nhdr->n_namesz == sizeof("GNU") &&
-		    nhdr->n_descsz > 0 &&
-		    nhdr->n_descsz <= BPF_BUILD_ID_SIZE) {
-			memcpy(build_id,
-			       note_start + note_offs +
-			       ALIGN(sizeof("GNU"), 4) + sizeof(Elf32_Nhdr),
-			       nhdr->n_descsz);
-			memset(build_id + nhdr->n_descsz, 0,
-			       BPF_BUILD_ID_SIZE - nhdr->n_descsz);
-			return 0;
-		}
-		new_offs = note_offs + sizeof(Elf32_Nhdr) +
-			ALIGN(nhdr->n_namesz, 4) + ALIGN(nhdr->n_descsz, 4);
-		if (new_offs <= note_offs)  /* overflow */
-			break;
-		note_offs = new_offs;
-	}
-	return -EINVAL;
-}
-
-/* Parse build ID from 32-bit ELF */
-static int stack_map_get_build_id_32(void *page_addr,
-				     unsigned char *build_id)
-{
-	Elf32_Ehdr *ehdr = (Elf32_Ehdr *)page_addr;
-	Elf32_Phdr *phdr;
-	int i;
-
-	/* only supports phdr that fits in one page */
-	if (ehdr->e_phnum >
-	    (PAGE_SIZE - sizeof(Elf32_Ehdr)) / sizeof(Elf32_Phdr))
-		return -EINVAL;
-
-	phdr = (Elf32_Phdr *)(page_addr + sizeof(Elf32_Ehdr));
-
-	for (i = 0; i < ehdr->e_phnum; ++i) {
-		if (phdr[i].p_type == PT_NOTE &&
-		    !stack_map_parse_build_id(page_addr, build_id,
-					      page_addr + phdr[i].p_offset,
-					      phdr[i].p_filesz))
-			return 0;
-	}
-	return -EINVAL;
-}
-
-/* Parse build ID from 64-bit ELF */
-static int stack_map_get_build_id_64(void *page_addr,
-				     unsigned char *build_id)
-{
-	Elf64_Ehdr *ehdr = (Elf64_Ehdr *)page_addr;
-	Elf64_Phdr *phdr;
-	int i;
-
-	/* only supports phdr that fits in one page */
-	if (ehdr->e_phnum >
-	    (PAGE_SIZE - sizeof(Elf64_Ehdr)) / sizeof(Elf64_Phdr))
-		return -EINVAL;
-
-	phdr = (Elf64_Phdr *)(page_addr + sizeof(Elf64_Ehdr));
-
-	for (i = 0; i < ehdr->e_phnum; ++i) {
-		if (phdr[i].p_type == PT_NOTE &&
-		    !stack_map_parse_build_id(page_addr, build_id,
-					      page_addr + phdr[i].p_offset,
-					      phdr[i].p_filesz))
-			return 0;
-	}
-	return -EINVAL;
-}
-
-/* Parse build ID of ELF file mapped to vma */
-static int stack_map_get_build_id(struct vm_area_struct *vma,
-				  unsigned char *build_id)
-{
-	Elf32_Ehdr *ehdr;
-	struct page *page;
-	void *page_addr;
-	int ret;
-
-	/* only works for page backed storage  */
-	if (!vma->vm_file)
-		return -EINVAL;
-
-	page = find_get_page(vma->vm_file->f_mapping, 0);
-	if (!page)
-		return -EFAULT;	/* page not mapped */
-
-	ret = -EINVAL;
-	page_addr = kmap_atomic(page);
-	ehdr = (Elf32_Ehdr *)page_addr;
-
-	/* compare magic x7f "ELF" */
-	if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) != 0)
-		goto out;
-
-	/* only support executable file and shared object file */
-	if (ehdr->e_type != ET_EXEC && ehdr->e_type != ET_DYN)
-		goto out;
-
-	if (ehdr->e_ident[EI_CLASS] == ELFCLASS32)
-		ret = stack_map_get_build_id_32(page_addr, build_id);
-	else if (ehdr->e_ident[EI_CLASS] == ELFCLASS64)
-		ret = stack_map_get_build_id_64(page_addr, build_id);
-out:
-	kunmap_atomic(page_addr);
-	put_page(page);
-	return ret;
-}
-
 static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
 					  u64 *ips, u32 trace_nr, bool user)
 {
@@ -327,18 +192,18 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
 		for (i = 0; i < trace_nr; i++) {
 			id_offs[i].status = BPF_STACK_BUILD_ID_IP;
 			id_offs[i].ip = ips[i];
-			memset(id_offs[i].build_id, 0, BPF_BUILD_ID_SIZE);
+			memset(id_offs[i].build_id, 0, BUILD_ID_SIZE);
 		}
 		return;
 	}
 
 	for (i = 0; i < trace_nr; i++) {
 		vma = find_vma(current->mm, ips[i]);
-		if (!vma || stack_map_get_build_id(vma, id_offs[i].build_id)) {
+		if (!vma || build_id_parse(vma, id_offs[i].build_id)) {
 			/* per entry fall back to ips */
 			id_offs[i].status = BPF_STACK_BUILD_ID_IP;
 			id_offs[i].ip = ips[i];
-			memset(id_offs[i].build_id, 0, BPF_BUILD_ID_SIZE);
+			memset(id_offs[i].build_id, 0, BUILD_ID_SIZE);
 			continue;
 		}
 		id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ips[i]
diff --git a/lib/Makefile b/lib/Makefile
index ce45af50983a..f4858f5e9215 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -36,7 +36,8 @@ lib-y := ctype.o string.o vsprintf.o cmdline.o \
 	 flex_proportions.o ratelimit.o show_mem.o \
 	 is_single_threaded.o plist.o decompress.o kobject_uevent.o \
 	 earlycpio.o seq_buf.o siphash.o dec_and_lock.o \
-	 nmi_backtrace.o nodemask.o win_minmax.o memcat_p.o
+	 nmi_backtrace.o nodemask.o win_minmax.o memcat_p.o \
+	 buildid.o
 
 lib-$(CONFIG_PRINTK) += dump_stack.o
 lib-$(CONFIG_SMP) += cpumask.o
diff --git a/lib/buildid.c b/lib/buildid.c
new file mode 100644
index 000000000000..e8d5feb7ef20
--- /dev/null
+++ b/lib/buildid.c
@@ -0,0 +1,136 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/buildid.h>
+#include <linux/elf.h>
+#include <linux/pagemap.h>
+
+#define BUILD_ID 3
+/*
+ * Parse build id from the note segment. This logic can be shared between
+ * 32-bit and 64-bit system, because Elf32_Nhdr and Elf64_Nhdr are
+ * identical.
+ */
+static inline int parse_build_id(void *page_addr,
+				 unsigned char *build_id,
+				 void *note_start,
+				 Elf32_Word note_size)
+{
+	Elf32_Word note_offs = 0, new_offs;
+
+	/* check for overflow */
+	if (note_start < page_addr || note_start + note_size < note_start)
+		return -EINVAL;
+
+	/* only supports note that fits in the first page */
+	if (note_start + note_size > page_addr + PAGE_SIZE)
+		return -EINVAL;
+
+	while (note_offs + sizeof(Elf32_Nhdr) < note_size) {
+		Elf32_Nhdr *nhdr = (Elf32_Nhdr *)(note_start + note_offs);
+
+		if (nhdr->n_type == BUILD_ID &&
+		    nhdr->n_namesz == sizeof("GNU") &&
+		    nhdr->n_descsz > 0 &&
+		    nhdr->n_descsz <= BUILD_ID_SIZE) {
+			memcpy(build_id,
+			       note_start + note_offs +
+			       ALIGN(sizeof("GNU"), 4) + sizeof(Elf32_Nhdr),
+			       nhdr->n_descsz);
+			memset(build_id + nhdr->n_descsz, 0,
+			       BUILD_ID_SIZE - nhdr->n_descsz);
+			return 0;
+		}
+		new_offs = note_offs + sizeof(Elf32_Nhdr) +
+			ALIGN(nhdr->n_namesz, 4) + ALIGN(nhdr->n_descsz, 4);
+		if (new_offs <= note_offs)  /* overflow */
+			break;
+		note_offs = new_offs;
+	}
+	return -EINVAL;
+}
+
+/* Parse build ID from 32-bit ELF */
+static int get_build_id_32(void *page_addr, unsigned char *build_id)
+{
+	Elf32_Ehdr *ehdr = (Elf32_Ehdr *)page_addr;
+	Elf32_Phdr *phdr;
+	int i;
+
+	/* only supports phdr that fits in one page */
+	if (ehdr->e_phnum >
+	    (PAGE_SIZE - sizeof(Elf32_Ehdr)) / sizeof(Elf32_Phdr))
+		return -EINVAL;
+
+	phdr = (Elf32_Phdr *)(page_addr + sizeof(Elf32_Ehdr));
+
+	for (i = 0; i < ehdr->e_phnum; ++i) {
+		if (phdr[i].p_type == PT_NOTE &&
+		    !parse_build_id(page_addr, build_id,
+				    page_addr + phdr[i].p_offset,
+				    phdr[i].p_filesz))
+			return 0;
+	}
+	return -EINVAL;
+}
+
+/* Parse build ID from 64-bit ELF */
+static int get_build_id_64(void *page_addr, unsigned char *build_id)
+{
+	Elf64_Ehdr *ehdr = (Elf64_Ehdr *)page_addr;
+	Elf64_Phdr *phdr;
+	int i;
+
+	/* only supports phdr that fits in one page */
+	if (ehdr->e_phnum >
+	    (PAGE_SIZE - sizeof(Elf64_Ehdr)) / sizeof(Elf64_Phdr))
+		return -EINVAL;
+
+	phdr = (Elf64_Phdr *)(page_addr + sizeof(Elf64_Ehdr));
+
+	for (i = 0; i < ehdr->e_phnum; ++i) {
+		if (phdr[i].p_type == PT_NOTE &&
+		    !parse_build_id(page_addr, build_id,
+				    page_addr + phdr[i].p_offset,
+				    phdr[i].p_filesz))
+			return 0;
+	}
+	return -EINVAL;
+}
+
+/* Parse build ID of ELF file mapped to vma */
+int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id)
+{
+	Elf32_Ehdr *ehdr;
+	struct page *page;
+	void *page_addr;
+	int ret;
+
+	/* only works for page backed storage  */
+	if (!vma->vm_file)
+		return -EINVAL;
+
+	page = find_get_page(vma->vm_file->f_mapping, 0);
+	if (!page)
+		return -EFAULT;	/* page not mapped */
+
+	ret = -EINVAL;
+	page_addr = kmap_atomic(page);
+	ehdr = (Elf32_Ehdr *)page_addr;
+
+	/* compare magic x7f "ELF" */
+	if (memcmp(ehdr->e_ident, ELFMAG, SELFMAG) != 0)
+		goto out;
+
+	/* only support executable file and shared object file */
+	if (ehdr->e_type != ET_EXEC && ehdr->e_type != ET_DYN)
+		goto out;
+
+	if (ehdr->e_ident[EI_CLASS] == ELFCLASS32)
+		ret = get_build_id_32(page_addr, build_id);
+	else if (ehdr->e_ident[EI_CLASS] == ELFCLASS64)
+		ret = get_build_id_64(page_addr, build_id);
+out:
+	kunmap_atomic(page_addr);
+	put_page(page);
+	return ret;
+}
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 02/25] bpf: Add size arg to build_id_parse function
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
  2020-11-23 23:04 ` [PATCH 01/25] bpf: Move stack_map_get_build_id into lib Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 03/25] perf: Add build id data in mmap2 event Jiri Olsa
                   ` (23 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Alexei Starovoitov, Song Liu, lkml, Peter Zijlstra, Ingo Molnar,
	Mark Rutland, Namhyung Kim, Alexander Shishkin, Michael Petlan,
	Ian Rogers, Stephane Eranian, Alexey Budankov, Andi Kleen,
	Adrian Hunter

It's possible to have other build id types (other than default SHA1).
Currently there's also ld support for MD5 build id.

Adding size argument to build_id_parse function, that returns (if defined)
size of the parsed build id, so we can recognize the build id type.

Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Song Liu <songliubraving@fb.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 include/linux/buildid.h |  3 ++-
 kernel/bpf/stackmap.c   |  2 +-
 lib/buildid.c           | 29 +++++++++++++++++++++--------
 3 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/include/linux/buildid.h b/include/linux/buildid.h
index 3be5b49719f1..72639d433873 100644
--- a/include/linux/buildid.h
+++ b/include/linux/buildid.h
@@ -6,6 +6,7 @@
 
 #define BUILD_ID_SIZE 20
 
-int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id);
+int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id,
+		   __u32 *size);
 
 #endif
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
index 7df08f8af5a1..a05fac2cbb05 100644
--- a/kernel/bpf/stackmap.c
+++ b/kernel/bpf/stackmap.c
@@ -199,7 +199,7 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
 
 	for (i = 0; i < trace_nr; i++) {
 		vma = find_vma(current->mm, ips[i]);
-		if (!vma || build_id_parse(vma, id_offs[i].build_id)) {
+		if (!vma || build_id_parse(vma, id_offs[i].build_id, NULL)) {
 			/* per entry fall back to ips */
 			id_offs[i].status = BPF_STACK_BUILD_ID_IP;
 			id_offs[i].ip = ips[i];
diff --git a/lib/buildid.c b/lib/buildid.c
index e8d5feb7ef20..dbe1b7e247f3 100644
--- a/lib/buildid.c
+++ b/lib/buildid.c
@@ -12,6 +12,7 @@
  */
 static inline int parse_build_id(void *page_addr,
 				 unsigned char *build_id,
+				 __u32 *size,
 				 void *note_start,
 				 Elf32_Word note_size)
 {
@@ -38,6 +39,8 @@ static inline int parse_build_id(void *page_addr,
 			       nhdr->n_descsz);
 			memset(build_id + nhdr->n_descsz, 0,
 			       BUILD_ID_SIZE - nhdr->n_descsz);
+			if (size)
+				*size = nhdr->n_descsz;
 			return 0;
 		}
 		new_offs = note_offs + sizeof(Elf32_Nhdr) +
@@ -50,7 +53,8 @@ static inline int parse_build_id(void *page_addr,
 }
 
 /* Parse build ID from 32-bit ELF */
-static int get_build_id_32(void *page_addr, unsigned char *build_id)
+static int get_build_id_32(void *page_addr, unsigned char *build_id,
+			   __u32 *size)
 {
 	Elf32_Ehdr *ehdr = (Elf32_Ehdr *)page_addr;
 	Elf32_Phdr *phdr;
@@ -65,7 +69,7 @@ static int get_build_id_32(void *page_addr, unsigned char *build_id)
 
 	for (i = 0; i < ehdr->e_phnum; ++i) {
 		if (phdr[i].p_type == PT_NOTE &&
-		    !parse_build_id(page_addr, build_id,
+		    !parse_build_id(page_addr, build_id, size,
 				    page_addr + phdr[i].p_offset,
 				    phdr[i].p_filesz))
 			return 0;
@@ -74,7 +78,8 @@ static int get_build_id_32(void *page_addr, unsigned char *build_id)
 }
 
 /* Parse build ID from 64-bit ELF */
-static int get_build_id_64(void *page_addr, unsigned char *build_id)
+static int get_build_id_64(void *page_addr, unsigned char *build_id,
+			   __u32 *size)
 {
 	Elf64_Ehdr *ehdr = (Elf64_Ehdr *)page_addr;
 	Elf64_Phdr *phdr;
@@ -89,7 +94,7 @@ static int get_build_id_64(void *page_addr, unsigned char *build_id)
 
 	for (i = 0; i < ehdr->e_phnum; ++i) {
 		if (phdr[i].p_type == PT_NOTE &&
-		    !parse_build_id(page_addr, build_id,
+		    !parse_build_id(page_addr, build_id, size,
 				    page_addr + phdr[i].p_offset,
 				    phdr[i].p_filesz))
 			return 0;
@@ -97,8 +102,16 @@ static int get_build_id_64(void *page_addr, unsigned char *build_id)
 	return -EINVAL;
 }
 
-/* Parse build ID of ELF file mapped to vma */
-int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id)
+/*
+ * Parse build ID of ELF file mapped to vma
+ * @vma:      vma object
+ * @build_id: buffer to store build id, at least BUILD_ID_SIZE long
+ * @size:     returns actual build id size in case of success
+ *
+ * Returns 0 on success, otherwise error (< 0).
+ */
+int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id,
+		   __u32 *size)
 {
 	Elf32_Ehdr *ehdr;
 	struct page *page;
@@ -126,9 +139,9 @@ int build_id_parse(struct vm_area_struct *vma, unsigned char *build_id)
 		goto out;
 
 	if (ehdr->e_ident[EI_CLASS] == ELFCLASS32)
-		ret = get_build_id_32(page_addr, build_id);
+		ret = get_build_id_32(page_addr, build_id, size);
 	else if (ehdr->e_ident[EI_CLASS] == ELFCLASS64)
-		ret = get_build_id_64(page_addr, build_id);
+		ret = get_build_id_64(page_addr, build_id, size);
 out:
 	kunmap_atomic(page_addr);
 	put_page(page);
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 03/25] perf: Add build id data in mmap2 event
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
  2020-11-23 23:04 ` [PATCH 01/25] bpf: Move stack_map_get_build_id into lib Jiri Olsa
  2020-11-23 23:04 ` [PATCH 02/25] bpf: Add size arg to build_id_parse function Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 04/25] tools headers uapi: Sync tools/include/uapi/linux/perf_event.h Jiri Olsa
                   ` (22 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Peter Zijlstra, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Ian Rogers, Stephane Eranian, Alexey Budankov, Andi Kleen,
	Adrian Hunter

Adding support to carry build id data in mmap2 event.

The build id data replaces maj/min/ino/ino_generation
fields, which are also used to identify map's binary,
so it's ok to replace them with build id data:

  union {
          struct {
                  u32       maj;
                  u32       min;
                  u64       ino;
                  u64       ino_generation;
          };
          struct {
                  u8        build_id_size;
                  u8        __reserved_1;
                  u16       __reserved_2;
                  u8        build_id[20];
          };
  };

Replaced maj/min/ino/ino_generation fields give us size
of 24 bytes. We use 20 bytes for build id data, 1 byte
for size and rest is unused.

There's new misc bit for mmap2 to signal there's build
id data in it:

  #define PERF_RECORD_MISC_MMAP_BUILD_ID   (1 << 14)

Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 include/uapi/linux/perf_event.h | 42 +++++++++++++++++++++++++++++----
 kernel/events/core.c            | 32 +++++++++++++++++++++----
 2 files changed, 65 insertions(+), 9 deletions(-)

diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index b95d3c485d27..45a216bea048 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -384,7 +384,8 @@ struct perf_event_attr {
 				aux_output     :  1, /* generate AUX records instead of events */
 				cgroup         :  1, /* include cgroup events */
 				text_poke      :  1, /* include text poke events */
-				__reserved_1   : 30;
+				build_id       :  1, /* use build id in mmap2 events */
+				__reserved_1   : 29;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */
@@ -657,6 +658,22 @@ struct perf_event_mmap_page {
 	__u64	aux_size;
 };
 
+/*
+ * The current state of perf_event_header::misc bits usage:
+ * ('|' used bit, '-' unused bit)
+ *
+ *  012         CDEF
+ *  |||---------||||
+ *
+ *  Where:
+ *    0-2     CPUMODE_MASK
+ *
+ *    C       PROC_MAP_PARSE_TIMEOUT
+ *    D       MMAP_DATA / COMM_EXEC / FORK_EXEC / SWITCH_OUT
+ *    E       MMAP_BUILD_ID / EXACT_IP / SCHED_OUT_PREEMPT
+ *    F       (reserved)
+ */
+
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
 #define PERF_RECORD_MISC_CPUMODE_UNKNOWN	(0 << 0)
 #define PERF_RECORD_MISC_KERNEL			(1 << 0)
@@ -688,6 +705,7 @@ struct perf_event_mmap_page {
  *
  *   PERF_RECORD_MISC_EXACT_IP           - PERF_RECORD_SAMPLE of precise events
  *   PERF_RECORD_MISC_SWITCH_OUT_PREEMPT - PERF_RECORD_SWITCH* events
+ *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event
  *
  *
  * PERF_RECORD_MISC_EXACT_IP:
@@ -697,9 +715,13 @@ struct perf_event_mmap_page {
  *
  * PERF_RECORD_MISC_SWITCH_OUT_PREEMPT:
  *   Indicates that thread was preempted in TASK_RUNNING state.
+ *
+ * PERF_RECORD_MISC_MMAP_BUILD_ID:
+ *   Indicates that mmap2 event carries build id data.
  */
 #define PERF_RECORD_MISC_EXACT_IP		(1 << 14)
 #define PERF_RECORD_MISC_SWITCH_OUT_PREEMPT	(1 << 14)
+#define PERF_RECORD_MISC_MMAP_BUILD_ID		(1 << 14)
 /*
  * Reserve the last bit to indicate some extended misc field
  */
@@ -911,10 +933,20 @@ enum perf_event_type {
 	 *	u64				addr;
 	 *	u64				len;
 	 *	u64				pgoff;
-	 *	u32				maj;
-	 *	u32				min;
-	 *	u64				ino;
-	 *	u64				ino_generation;
+	 *	union {
+	 *		struct {
+	 *			u32		maj;
+	 *			u32		min;
+	 *			u64		ino;
+	 *			u64		ino_generation;
+	 *		};
+	 *		struct {
+	 *			u8		build_id_size;
+	 *			u8		__reserved_1;
+	 *			u16		__reserved_2;
+	 *			u8		build_id[20];
+	 *		};
+	 *	};
 	 *	u32				prot, flags;
 	 *	char				filename[];
 	 * 	struct sample_id		sample_id;
diff --git a/kernel/events/core.c b/kernel/events/core.c
index da467e1dd49a..5841b5bca68d 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -51,6 +51,7 @@
 #include <linux/proc_ns.h>
 #include <linux/mount.h>
 #include <linux/min_heap.h>
+#include <linux/buildid.h>
 
 #include "internal.h"
 
@@ -395,6 +396,7 @@ static atomic_t nr_ksymbol_events __read_mostly;
 static atomic_t nr_bpf_events __read_mostly;
 static atomic_t nr_cgroup_events __read_mostly;
 static atomic_t nr_text_poke_events __read_mostly;
+static atomic_t nr_build_id_events __read_mostly;
 
 static LIST_HEAD(pmus);
 static DEFINE_MUTEX(pmus_lock);
@@ -4672,6 +4674,8 @@ static void unaccount_event(struct perf_event *event)
 		dec = true;
 	if (event->attr.mmap || event->attr.mmap_data)
 		atomic_dec(&nr_mmap_events);
+	if (event->attr.build_id)
+		atomic_dec(&nr_build_id_events);
 	if (event->attr.comm)
 		atomic_dec(&nr_comm_events);
 	if (event->attr.namespaces)
@@ -7942,6 +7946,8 @@ struct perf_mmap_event {
 	u64			ino;
 	u64			ino_generation;
 	u32			prot, flags;
+	u8			build_id[BUILD_ID_SIZE];
+	u32			build_id_size;
 
 	struct {
 		struct perf_event_header	header;
@@ -7973,6 +7979,7 @@ static void perf_event_mmap_output(struct perf_event *event,
 	struct perf_sample_data sample;
 	int size = mmap_event->event_id.header.size;
 	u32 type = mmap_event->event_id.header.type;
+	bool use_build_id;
 	int ret;
 
 	if (!perf_event_mmap_match(event, data))
@@ -7997,13 +8004,25 @@ static void perf_event_mmap_output(struct perf_event *event,
 	mmap_event->event_id.pid = perf_event_pid(event, current);
 	mmap_event->event_id.tid = perf_event_tid(event, current);
 
+	use_build_id = event->attr.build_id && mmap_event->build_id_size;
+
+	if (event->attr.mmap2 && use_build_id)
+		mmap_event->event_id.header.misc |= PERF_RECORD_MISC_MMAP_BUILD_ID;
+
 	perf_output_put(&handle, mmap_event->event_id);
 
 	if (event->attr.mmap2) {
-		perf_output_put(&handle, mmap_event->maj);
-		perf_output_put(&handle, mmap_event->min);
-		perf_output_put(&handle, mmap_event->ino);
-		perf_output_put(&handle, mmap_event->ino_generation);
+		if (use_build_id) {
+			u8 size[4] = { (u8) mmap_event->build_id_size, 0, 0, 0 };
+
+			__output_copy(&handle, size, 4);
+			__output_copy(&handle, mmap_event->build_id, BUILD_ID_SIZE);
+		} else {
+			perf_output_put(&handle, mmap_event->maj);
+			perf_output_put(&handle, mmap_event->min);
+			perf_output_put(&handle, mmap_event->ino);
+			perf_output_put(&handle, mmap_event->ino_generation);
+		}
 		perf_output_put(&handle, mmap_event->prot);
 		perf_output_put(&handle, mmap_event->flags);
 	}
@@ -8132,6 +8151,9 @@ static void perf_event_mmap_event(struct perf_mmap_event *mmap_event)
 
 	mmap_event->event_id.header.size = sizeof(mmap_event->event_id) + size;
 
+	if (atomic_read(&nr_build_id_events))
+		build_id_parse(vma, mmap_event->build_id, &mmap_event->build_id_size);
+
 	perf_iterate_sb(perf_event_mmap_output,
 		       mmap_event,
 		       NULL);
@@ -11069,6 +11091,8 @@ static void account_event(struct perf_event *event)
 		inc = true;
 	if (event->attr.mmap || event->attr.mmap_data)
 		atomic_inc(&nr_mmap_events);
+	if (event->attr.build_id)
+		atomic_inc(&nr_build_id_events);
 	if (event->attr.comm)
 		atomic_inc(&nr_comm_events);
 	if (event->attr.namespaces)
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 04/25] tools headers uapi: Sync tools/include/uapi/linux/perf_event.h
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (2 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 03/25] perf: Add build id data in mmap2 event Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 05/25] tools lib: Adopt memchr_inv() from kernel Jiri Olsa
                   ` (21 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Syncing tools's uapi with mmap2 build id data changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/include/uapi/linux/perf_event.h | 42 +++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 5 deletions(-)

diff --git a/tools/include/uapi/linux/perf_event.h b/tools/include/uapi/linux/perf_event.h
index b95d3c485d27..45a216bea048 100644
--- a/tools/include/uapi/linux/perf_event.h
+++ b/tools/include/uapi/linux/perf_event.h
@@ -384,7 +384,8 @@ struct perf_event_attr {
 				aux_output     :  1, /* generate AUX records instead of events */
 				cgroup         :  1, /* include cgroup events */
 				text_poke      :  1, /* include text poke events */
-				__reserved_1   : 30;
+				build_id       :  1, /* use build id in mmap2 events */
+				__reserved_1   : 29;
 
 	union {
 		__u32		wakeup_events;	  /* wakeup every n events */
@@ -657,6 +658,22 @@ struct perf_event_mmap_page {
 	__u64	aux_size;
 };
 
+/*
+ * The current state of perf_event_header::misc bits usage:
+ * ('|' used bit, '-' unused bit)
+ *
+ *  012         CDEF
+ *  |||---------||||
+ *
+ *  Where:
+ *    0-2     CPUMODE_MASK
+ *
+ *    C       PROC_MAP_PARSE_TIMEOUT
+ *    D       MMAP_DATA / COMM_EXEC / FORK_EXEC / SWITCH_OUT
+ *    E       MMAP_BUILD_ID / EXACT_IP / SCHED_OUT_PREEMPT
+ *    F       (reserved)
+ */
+
 #define PERF_RECORD_MISC_CPUMODE_MASK		(7 << 0)
 #define PERF_RECORD_MISC_CPUMODE_UNKNOWN	(0 << 0)
 #define PERF_RECORD_MISC_KERNEL			(1 << 0)
@@ -688,6 +705,7 @@ struct perf_event_mmap_page {
  *
  *   PERF_RECORD_MISC_EXACT_IP           - PERF_RECORD_SAMPLE of precise events
  *   PERF_RECORD_MISC_SWITCH_OUT_PREEMPT - PERF_RECORD_SWITCH* events
+ *   PERF_RECORD_MISC_MMAP_BUILD_ID      - PERF_RECORD_MMAP2 event
  *
  *
  * PERF_RECORD_MISC_EXACT_IP:
@@ -697,9 +715,13 @@ struct perf_event_mmap_page {
  *
  * PERF_RECORD_MISC_SWITCH_OUT_PREEMPT:
  *   Indicates that thread was preempted in TASK_RUNNING state.
+ *
+ * PERF_RECORD_MISC_MMAP_BUILD_ID:
+ *   Indicates that mmap2 event carries build id data.
  */
 #define PERF_RECORD_MISC_EXACT_IP		(1 << 14)
 #define PERF_RECORD_MISC_SWITCH_OUT_PREEMPT	(1 << 14)
+#define PERF_RECORD_MISC_MMAP_BUILD_ID		(1 << 14)
 /*
  * Reserve the last bit to indicate some extended misc field
  */
@@ -911,10 +933,20 @@ enum perf_event_type {
 	 *	u64				addr;
 	 *	u64				len;
 	 *	u64				pgoff;
-	 *	u32				maj;
-	 *	u32				min;
-	 *	u64				ino;
-	 *	u64				ino_generation;
+	 *	union {
+	 *		struct {
+	 *			u32		maj;
+	 *			u32		min;
+	 *			u64		ino;
+	 *			u64		ino_generation;
+	 *		};
+	 *		struct {
+	 *			u8		build_id_size;
+	 *			u8		__reserved_1;
+	 *			u16		__reserved_2;
+	 *			u8		build_id[20];
+	 *		};
+	 *	};
 	 *	u32				prot, flags;
 	 *	char				filename[];
 	 * 	struct sample_id		sample_id;
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 05/25] tools lib: Adopt memchr_inv() from kernel
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (3 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 04/25] tools headers uapi: Sync tools/include/uapi/linux/perf_event.h Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 06/25] perf tools: Do not swap mmap2 fields in case it contains build id Jiri Olsa
                   ` (20 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

We'll use it to check for undefined/zero data.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/include/linux/string.h |  1 +
 tools/lib/string.c           | 58 ++++++++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+)

diff --git a/tools/include/linux/string.h b/tools/include/linux/string.h
index 5e9e781905ed..db5c99318c79 100644
--- a/tools/include/linux/string.h
+++ b/tools/include/linux/string.h
@@ -46,4 +46,5 @@ extern char * __must_check skip_spaces(const char *);
 
 extern char *strim(char *);
 
+extern void *memchr_inv(const void *start, int c, size_t bytes);
 #endif /* _TOOLS_LINUX_STRING_H_ */
diff --git a/tools/lib/string.c b/tools/lib/string.c
index f645343815de..8b6892f959ab 100644
--- a/tools/lib/string.c
+++ b/tools/lib/string.c
@@ -168,3 +168,61 @@ char *strreplace(char *s, char old, char new)
 			*s = new;
 	return s;
 }
+
+static void *check_bytes8(const u8 *start, u8 value, unsigned int bytes)
+{
+	while (bytes) {
+		if (*start != value)
+			return (void *)start;
+		start++;
+		bytes--;
+	}
+	return NULL;
+}
+
+/**
+ * memchr_inv - Find an unmatching character in an area of memory.
+ * @start: The memory area
+ * @c: Find a character other than c
+ * @bytes: The size of the area.
+ *
+ * returns the address of the first character other than @c, or %NULL
+ * if the whole buffer contains just @c.
+ */
+void *memchr_inv(const void *start, int c, size_t bytes)
+{
+	u8 value = c;
+	u64 value64;
+	unsigned int words, prefix;
+
+	if (bytes <= 16)
+		return check_bytes8(start, value, bytes);
+
+	value64 = value;
+	value64 |= value64 << 8;
+	value64 |= value64 << 16;
+	value64 |= value64 << 32;
+
+	prefix = (unsigned long)start % 8;
+	if (prefix) {
+		u8 *r;
+
+		prefix = 8 - prefix;
+		r = check_bytes8(start, value, prefix);
+		if (r)
+			return r;
+		start += prefix;
+		bytes -= prefix;
+	}
+
+	words = bytes / 8;
+
+	while (words) {
+		if (*(u64 *)start != value64)
+			return check_bytes8(start, value, 8);
+		start += 8;
+		words--;
+	}
+
+	return check_bytes8(start, value, bytes % 8);
+}
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 06/25] perf tools: Do not swap mmap2 fields in case it contains build id
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (4 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 05/25] tools lib: Adopt memchr_inv() from kernel Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 07/25] perf tools: Add build_id__is_defined function Jiri Olsa
                   ` (19 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

If PERF_RECORD_MISC_MMAP_BUILD_ID misc bit is set,
mmap2 event carries build id, placed in following union:

  union {
          struct {
                  u32       maj;
                  u32       min;
                  u64       ino;
                  u64       ino_generation;
          };
          struct {
                  u8        build_id[20];
                  u8        build_id_size;
                  u8        __reserved_1;
                  u16       __reserved_2;
          };
  };

In this case we can't swap above fields.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/session.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/tools/perf/util/session.c b/tools/perf/util/session.c
index 5cc722b6fe7c..cc1c11ca94fd 100644
--- a/tools/perf/util/session.c
+++ b/tools/perf/util/session.c
@@ -592,10 +592,13 @@ static void perf_event__mmap2_swap(union perf_event *event,
 	event->mmap2.start = bswap_64(event->mmap2.start);
 	event->mmap2.len   = bswap_64(event->mmap2.len);
 	event->mmap2.pgoff = bswap_64(event->mmap2.pgoff);
-	event->mmap2.maj   = bswap_32(event->mmap2.maj);
-	event->mmap2.min   = bswap_32(event->mmap2.min);
-	event->mmap2.ino   = bswap_64(event->mmap2.ino);
-	event->mmap2.ino_generation = bswap_64(event->mmap2.ino_generation);
+
+	if (!(event->header.misc & PERF_RECORD_MISC_MMAP_BUILD_ID)) {
+		event->mmap2.maj   = bswap_32(event->mmap2.maj);
+		event->mmap2.min   = bswap_32(event->mmap2.min);
+		event->mmap2.ino   = bswap_64(event->mmap2.ino);
+		event->mmap2.ino_generation = bswap_64(event->mmap2.ino_generation);
+	}
 
 	if (sample_id_all) {
 		void *data = &event->mmap2.filename;
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 07/25] perf tools: Add build_id__is_defined function
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (5 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 06/25] perf tools: Do not swap mmap2 fields in case it contains build id Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 08/25] perf tools: Add filename__decompress function Jiri Olsa
                   ` (18 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding build_id__is_defined helper to check build id
is defined and is != zero build id.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 6 ++++++
 tools/perf/util/build-id.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 6b410c3d52dc..2aacc8b29f7e 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -37,6 +37,7 @@
 
 #include <linux/ctype.h>
 #include <linux/zalloc.h>
+#include <linux/string.h>
 #include <asm/bug.h>
 
 static bool no_buildid_cache;
@@ -912,3 +913,8 @@ void build_id__init(struct build_id *bid, const u8 *data, size_t size)
 	memcpy(bid->data, data, size);
 	bid->size = size;
 }
+
+bool build_id__is_defined(const struct build_id *bid)
+{
+	return bid && bid->size ? !!memchr_inv(bid->data, 0, bid->size) : false;
+}
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index f293f99d5dba..d53415feaf69 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -21,6 +21,7 @@ struct feat_fd;
 
 void build_id__init(struct build_id *bid, const u8 *data, size_t size);
 int build_id__sprintf(const struct build_id *build_id, char *bf);
+bool build_id__is_defined(const struct build_id *bid);
 int sysfs__sprintf_build_id(const char *root_dir, char *sbuild_id);
 int filename__sprintf_build_id(const char *pathname, char *sbuild_id);
 char *build_id_cache__kallsyms_path(const char *sbuild_id, char *bf,
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 08/25] perf tools: Add filename__decompress function
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (6 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 07/25] perf tools: Add build_id__is_defined function Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 09/25] perf tools: Add support to read build id from compressed elf Jiri Olsa
                   ` (17 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Factor filename__decompress from decompress_kmodule function.
It can decompress files with compressions supported in perf -
xz and gz, the support needs to be compiled in.

It will to be used in following changes to get build id out of
compressed elf objects.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/dso.c | 31 +++++++++++++++++++------------
 tools/perf/util/dso.h |  2 ++
 2 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 89b5fd2b5de3..d786cf6b0cfa 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -279,18 +279,12 @@ bool dso__needs_decompress(struct dso *dso)
 		dso->symtab_type == DSO_BINARY_TYPE__GUEST_KMODULE_COMP;
 }
 
-static int decompress_kmodule(struct dso *dso, const char *name,
-			      char *pathname, size_t len)
+int filename__decompress(const char *name, char *pathname,
+			 size_t len, int comp, int *err)
 {
 	char tmpbuf[] = KMOD_DECOMP_NAME;
 	int fd = -1;
 
-	if (!dso__needs_decompress(dso))
-		return -1;
-
-	if (dso->comp == COMP_ID__NONE)
-		return -1;
-
 	/*
 	 * We have proper compression id for DSO and yet the file
 	 * behind the 'name' can still be plain uncompressed object.
@@ -304,17 +298,17 @@ static int decompress_kmodule(struct dso *dso, const char *name,
 	 * To keep this transparent, we detect this and return the file
 	 * descriptor to the uncompressed file.
 	 */
-	if (!compressions[dso->comp].is_compressed(name))
+	if (!compressions[comp].is_compressed(name))
 		return open(name, O_RDONLY);
 
 	fd = mkstemp(tmpbuf);
 	if (fd < 0) {
-		dso->load_errno = errno;
+		*err = errno;
 		return -1;
 	}
 
-	if (compressions[dso->comp].decompress(name, fd)) {
-		dso->load_errno = DSO_LOAD_ERRNO__DECOMPRESSION_FAILURE;
+	if (compressions[comp].decompress(name, fd)) {
+		*err = DSO_LOAD_ERRNO__DECOMPRESSION_FAILURE;
 		close(fd);
 		fd = -1;
 	}
@@ -328,6 +322,19 @@ static int decompress_kmodule(struct dso *dso, const char *name,
 	return fd;
 }
 
+static int decompress_kmodule(struct dso *dso, const char *name,
+			      char *pathname, size_t len)
+{
+	if (!dso__needs_decompress(dso))
+		return -1;
+
+	if (dso->comp == COMP_ID__NONE)
+		return -1;
+
+	return filename__decompress(name, pathname, len, dso->comp,
+				    &dso->load_errno);
+}
+
 int dso__decompress_kmodule_fd(struct dso *dso, const char *name)
 {
 	return decompress_kmodule(dso, name, NULL, 0);
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index d8cb4f5680a4..cd2fe64a3c5d 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -274,6 +274,8 @@ bool dso__needs_decompress(struct dso *dso);
 int dso__decompress_kmodule_fd(struct dso *dso, const char *name);
 int dso__decompress_kmodule_path(struct dso *dso, const char *name,
 				 char *pathname, size_t len);
+int filename__decompress(const char *name, char *pathname,
+			 size_t len, int comp, int *err);
 
 #define KMOD_DECOMP_NAME  "/tmp/perf-kmod-XXXXXX"
 #define KMOD_DECOMP_LEN   sizeof(KMOD_DECOMP_NAME)
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 09/25] perf tools: Add support to read build id from compressed elf
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (7 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 08/25] perf tools: Add filename__decompress function Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 10/25] perf tools: Add check for existing link in buildid dir Jiri Olsa
                   ` (16 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding support to decompress file before reading build id.

Adding filename__read_build_id and change its current
versions to read_build_id.

Shutting down stderr output of perf list in the shell test:
  82: Check open filename arg using perf trace + vfs_getname          : Ok

because with decompression code in the place we the
filename__read_build_id function is more verbose in case
of error and the test did not account for that.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../tests/shell/trace+probe_vfs_getname.sh    |  2 +-
 tools/perf/util/symbol-elf.c                  | 37 ++++++++++++++++++-
 2 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/tools/perf/tests/shell/trace+probe_vfs_getname.sh b/tools/perf/tests/shell/trace+probe_vfs_getname.sh
index 11cc2af13f2b..3660fcc02fef 100755
--- a/tools/perf/tests/shell/trace+probe_vfs_getname.sh
+++ b/tools/perf/tests/shell/trace+probe_vfs_getname.sh
@@ -20,7 +20,7 @@ skip_if_no_perf_trace || exit 2
 file=$(mktemp /tmp/temporary_file.XXXXX)
 
 trace_open_vfs_getname() {
-	evts=$(echo $(perf list syscalls:sys_enter_open* 2>&1 | egrep 'open(at)? ' | sed -r 's/.*sys_enter_([a-z]+) +\[.*$/\1/') | sed 's/ /,/')
+	evts=$(echo $(perf list syscalls:sys_enter_open* >&1 2>/dev/nul | egrep 'open(at)? ' | sed -r 's/.*sys_enter_([a-z]+) +\[.*$/\1/') | sed 's/ /,/')
 	perf trace -e $evts touch $file 2>&1 | \
 	egrep " +[0-9]+\.[0-9]+ +\( +[0-9]+\.[0-9]+ ms\): +touch\/[0-9]+ open(at)?\((dfd: +CWD, +)?filename: +${file}, +flags: CREAT\|NOCTTY\|NONBLOCK\|WRONLY, +mode: +IRUGO\|IWUGO\) += +[0-9]+$"
 }
diff --git a/tools/perf/util/symbol-elf.c b/tools/perf/util/symbol-elf.c
index 44dd86a4f25f..f3577f7d72fe 100644
--- a/tools/perf/util/symbol-elf.c
+++ b/tools/perf/util/symbol-elf.c
@@ -534,7 +534,7 @@ static int elf_read_build_id(Elf *elf, void *bf, size_t size)
 
 #ifdef HAVE_LIBBFD_BUILDID_SUPPORT
 
-int filename__read_build_id(const char *filename, struct build_id *bid)
+static int read_build_id(const char *filename, struct build_id *bid)
 {
 	size_t size = sizeof(bid->data);
 	int err = -1;
@@ -563,7 +563,7 @@ int filename__read_build_id(const char *filename, struct build_id *bid)
 
 #else // HAVE_LIBBFD_BUILDID_SUPPORT
 
-int filename__read_build_id(const char *filename, struct build_id *bid)
+static int read_build_id(const char *filename, struct build_id *bid)
 {
 	size_t size = sizeof(bid->data);
 	int fd, err = -1;
@@ -595,6 +595,39 @@ int filename__read_build_id(const char *filename, struct build_id *bid)
 
 #endif // HAVE_LIBBFD_BUILDID_SUPPORT
 
+int filename__read_build_id(const char *filename, struct build_id *bid)
+{
+	struct kmod_path m = { .name = NULL, };
+	char path[PATH_MAX];
+	int err;
+
+	if (!filename)
+		return -EFAULT;
+
+	err = kmod_path__parse(&m, filename);
+	if (err)
+		return -1;
+
+	if (m.comp) {
+		int error = 0, fd;
+
+		fd = filename__decompress(filename, path, sizeof(path), m.comp, &error);
+		if (fd < 0) {
+			pr_debug("Failed to decompress (error %d) %s\n",
+				 error, filename);
+			return -1;
+		}
+		close(fd);
+		filename = path;
+	}
+
+	err = read_build_id(filename, bid);
+
+	if (m.comp)
+		unlink(filename);
+	return err;
+}
+
 int sysfs__read_build_id(const char *filename, struct build_id *bid)
 {
 	size_t size = sizeof(bid->data);
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 10/25] perf tools: Add check for existing link in buildid dir
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (8 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 09/25] perf tools: Add support to read build id from compressed elf Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 11/25] perf tools: Use struct extra_kernel_map in machine__process_kernel_mmap_event Jiri Olsa
                   ` (15 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

When adding new build id link we fail if the link is already
there. Adding check for existing link and output debug message
that the build id is already linked.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 2aacc8b29f7e..4a391f13f40d 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -755,8 +755,25 @@ int build_id_cache__add_s(const char *sbuild_id, const char *name,
 	tmp = dir_name + strlen(buildid_dir) - 5;
 	memcpy(tmp, "../..", 5);
 
-	if (symlink(tmp, linkname) == 0)
+	if (symlink(tmp, linkname) == 0) {
 		err = 0;
+	} else if (errno == EEXIST) {
+		char path[PATH_MAX];
+		ssize_t len;
+
+		len = readlink(linkname, path, sizeof(path) - 1);
+		if (len <= 0) {
+			pr_err("Cant read link: %s\n", linkname);
+			goto out_free;
+		}
+		path[len] = '\0';
+
+		if (strcmp(tmp, path)) {
+			pr_debug("build <%s> already linked to %s\n",
+				 sbuild_id, linkname);
+		}
+		err = 0;
+	}
 
 	/* Update SDT cache : error is just warned */
 	if (realname &&
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 11/25] perf tools: Use struct extra_kernel_map in machine__process_kernel_mmap_event
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (9 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 10/25] perf tools: Add check for existing link in buildid dir Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:04 ` [PATCH 12/25] perf tools: Try to load vmlinux from buildid database Jiri Olsa
                   ` (14 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ian Rogers, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Using struct extra_kernel_map in machine__process_kernel_mmap_event,
to pass mmap details. This way we can used single function for all 3
mmap versions.

Acked-by: Ian Rogers <irogers@google.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/machine.c | 62 +++++++++++++++++++++------------------
 1 file changed, 33 insertions(+), 29 deletions(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 15385ea00190..1ae32a81639c 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1581,32 +1581,25 @@ static bool machine__uses_kcore(struct machine *machine)
 }
 
 static bool perf_event__is_extra_kernel_mmap(struct machine *machine,
-					     union perf_event *event)
+					     struct extra_kernel_map *xm)
 {
 	return machine__is(machine, "x86_64") &&
-	       is_entry_trampoline(event->mmap.filename);
+	       is_entry_trampoline(xm->name);
 }
 
 static int machine__process_extra_kernel_map(struct machine *machine,
-					     union perf_event *event)
+					     struct extra_kernel_map *xm)
 {
 	struct dso *kernel = machine__kernel_dso(machine);
-	struct extra_kernel_map xm = {
-		.start = event->mmap.start,
-		.end   = event->mmap.start + event->mmap.len,
-		.pgoff = event->mmap.pgoff,
-	};
 
 	if (kernel == NULL)
 		return -1;
 
-	strlcpy(xm.name, event->mmap.filename, KMAP_NAME_LEN);
-
-	return machine__create_extra_kernel_map(machine, kernel, &xm);
+	return machine__create_extra_kernel_map(machine, kernel, xm);
 }
 
 static int machine__process_kernel_mmap_event(struct machine *machine,
-					      union perf_event *event)
+					      struct extra_kernel_map *xm)
 {
 	struct map *map;
 	enum dso_space_type dso_space;
@@ -1621,20 +1614,18 @@ static int machine__process_kernel_mmap_event(struct machine *machine,
 	else
 		dso_space = DSO_SPACE__KERNEL_GUEST;
 
-	is_kernel_mmap = memcmp(event->mmap.filename,
-				machine->mmap_name,
+	is_kernel_mmap = memcmp(xm->name, machine->mmap_name,
 				strlen(machine->mmap_name) - 1) == 0;
-	if (event->mmap.filename[0] == '/' ||
-	    (!is_kernel_mmap && event->mmap.filename[0] == '[')) {
-		map = machine__addnew_module_map(machine, event->mmap.start,
-						 event->mmap.filename);
+	if (xm->name[0] == '/' ||
+	    (!is_kernel_mmap && xm->name[0] == '[')) {
+		map = machine__addnew_module_map(machine, xm->start,
+						 xm->name);
 		if (map == NULL)
 			goto out_problem;
 
-		map->end = map->start + event->mmap.len;
+		map->end = map->start + xm->end - xm->start;
 	} else if (is_kernel_mmap) {
-		const char *symbol_name = (event->mmap.filename +
-				strlen(machine->mmap_name));
+		const char *symbol_name = (xm->name + strlen(machine->mmap_name));
 		/*
 		 * Should be there already, from the build-id table in
 		 * the header.
@@ -1688,18 +1679,17 @@ static int machine__process_kernel_mmap_event(struct machine *machine,
 		if (strstr(kernel->long_name, "vmlinux"))
 			dso__set_short_name(kernel, "[kernel.vmlinux]", false);
 
-		machine__update_kernel_mmap(machine, event->mmap.start,
-					 event->mmap.start + event->mmap.len);
+		machine__update_kernel_mmap(machine, xm->start, xm->end);
 
 		/*
 		 * Avoid using a zero address (kptr_restrict) for the ref reloc
 		 * symbol. Effectively having zero here means that at record
 		 * time /proc/sys/kernel/kptr_restrict was non zero.
 		 */
-		if (event->mmap.pgoff != 0) {
+		if (xm->pgoff != 0) {
 			map__set_kallsyms_ref_reloc_sym(machine->vmlinux_map,
 							symbol_name,
-							event->mmap.pgoff);
+							xm->pgoff);
 		}
 
 		if (machine__is_default_guest(machine)) {
@@ -1708,8 +1698,8 @@ static int machine__process_kernel_mmap_event(struct machine *machine,
 			 */
 			dso__load(kernel, machine__kernel_map(machine));
 		}
-	} else if (perf_event__is_extra_kernel_mmap(machine, event)) {
-		return machine__process_extra_kernel_map(machine, event);
+	} else if (perf_event__is_extra_kernel_mmap(machine, xm)) {
+		return machine__process_extra_kernel_map(machine, xm);
 	}
 	return 0;
 out_problem:
@@ -1735,7 +1725,14 @@ int machine__process_mmap2_event(struct machine *machine,
 
 	if (sample->cpumode == PERF_RECORD_MISC_GUEST_KERNEL ||
 	    sample->cpumode == PERF_RECORD_MISC_KERNEL) {
-		ret = machine__process_kernel_mmap_event(machine, event);
+		struct extra_kernel_map xm = {
+			.start = event->mmap2.start,
+			.end   = event->mmap2.start + event->mmap2.len,
+			.pgoff = event->mmap2.pgoff,
+		};
+
+		strlcpy(xm.name, event->mmap2.filename, KMAP_NAME_LEN);
+		ret = machine__process_kernel_mmap_event(machine, &xm);
 		if (ret < 0)
 			goto out_problem;
 		return 0;
@@ -1785,7 +1782,14 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
 	if (sample->cpumode == PERF_RECORD_MISC_GUEST_KERNEL ||
 	    sample->cpumode == PERF_RECORD_MISC_KERNEL) {
-		ret = machine__process_kernel_mmap_event(machine, event);
+		struct extra_kernel_map xm = {
+			.start = event->mmap.start,
+			.end   = event->mmap.start + event->mmap.len,
+			.pgoff = event->mmap.pgoff,
+		};
+
+		strlcpy(xm.name, event->mmap.filename, KMAP_NAME_LEN);
+		ret = machine__process_kernel_mmap_event(machine, &xm);
 		if (ret < 0)
 			goto out_problem;
 		return 0;
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 12/25] perf tools: Try to load vmlinux from buildid database
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (10 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 11/25] perf tools: Use struct extra_kernel_map in machine__process_kernel_mmap_event Jiri Olsa
@ 2020-11-23 23:04 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 13/25] perf tools: Store build id from mmap2 events Jiri Olsa
                   ` (13 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:04 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Currently we don't check on kernel's vmlinux the same way as
we do for normal binaries, but we either look for kallsyms
file in build id database or check on known vmlinux locations
(plus some other optional paths).

This patch adds the check for standard build id binary location,
so we are ready once we start to store it there from debuginfod
in following changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 13 ++++++++++---
 tools/perf/util/build-id.h |  2 ++
 tools/perf/util/symbol.c   | 16 ++++++++++++++++
 3 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 4a391f13f40d..1fd58703c2d4 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -261,10 +261,9 @@ static const char *build_id_cache__basename(bool is_kallsyms, bool is_vdso,
 	    "debug" : "elf"));
 }
 
-char *dso__build_id_filename(const struct dso *dso, char *bf, size_t size,
-			     bool is_debug)
+char *__dso__build_id_filename(const struct dso *dso, char *bf, size_t size,
+			       bool is_debug, bool is_kallsyms)
 {
-	bool is_kallsyms = dso__is_kallsyms((struct dso *)dso);
 	bool is_vdso = dso__is_vdso((struct dso *)dso);
 	char sbuild_id[SBUILD_ID_SIZE];
 	char *linkname;
@@ -293,6 +292,14 @@ char *dso__build_id_filename(const struct dso *dso, char *bf, size_t size,
 	return bf;
 }
 
+char *dso__build_id_filename(const struct dso *dso, char *bf, size_t size,
+			     bool is_debug)
+{
+	bool is_kallsyms = dso__is_kallsyms((struct dso *)dso);
+
+	return __dso__build_id_filename(dso, bf, size, is_debug, is_kallsyms);
+}
+
 #define dsos__for_each_with_build_id(pos, head)	\
 	list_for_each_entry(pos, head, node)	\
 		if (!pos->has_build_id)		\
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index d53415feaf69..f1a2f67df6e4 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -29,6 +29,8 @@ char *build_id_cache__kallsyms_path(const char *sbuild_id, char *bf,
 
 char *dso__build_id_filename(const struct dso *dso, char *bf, size_t size,
 			     bool is_debug);
+char *__dso__build_id_filename(const struct dso *dso, char *bf, size_t size,
+			       bool is_debug, bool is_kallsyms);
 
 int build_id__mark_dso_hit(struct perf_tool *tool, union perf_event *event,
 			   struct perf_sample *sample, struct evsel *evsel,
diff --git a/tools/perf/util/symbol.c b/tools/perf/util/symbol.c
index 0d14abdf3d72..64a039cbba1b 100644
--- a/tools/perf/util/symbol.c
+++ b/tools/perf/util/symbol.c
@@ -2189,6 +2189,8 @@ static int dso__load_kernel_sym(struct dso *dso, struct map *map)
 	int err;
 	const char *kallsyms_filename = NULL;
 	char *kallsyms_allocated_filename = NULL;
+	char *filename = NULL;
+
 	/*
 	 * Step 1: if the user specified a kallsyms or vmlinux filename, use
 	 * it and only it, reporting errors to the user if it cannot be used.
@@ -2213,6 +2215,20 @@ static int dso__load_kernel_sym(struct dso *dso, struct map *map)
 		return dso__load_vmlinux(dso, map, symbol_conf.vmlinux_name, false);
 	}
 
+	/*
+	 * Before checking on common vmlinux locations, check if it's
+	 * stored as standard build id binary (not kallsyms) under
+	 * .debug cache.
+	 */
+	if (!symbol_conf.ignore_vmlinux_buildid)
+		filename = __dso__build_id_filename(dso, NULL, 0, false, false);
+	if (filename != NULL) {
+		err = dso__load_vmlinux(dso, map, filename, true);
+		if (err > 0)
+			return err;
+		free(filename);
+	}
+
 	if (!symbol_conf.ignore_vmlinux && vmlinux_path != NULL) {
 		err = dso__load_vmlinux_path(dso, map);
 		if (err > 0)
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 13/25] perf tools: Store build id from mmap2 events
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (11 preceding siblings ...)
  2020-11-23 23:04 ` [PATCH 12/25] perf tools: Try to load vmlinux from buildid database Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-25 12:56   ` Namhyung Kim
  2020-11-23 23:05 ` [PATCH 14/25] perf tools: Allow mmap2 event to synthesize kernel image Jiri Olsa
                   ` (12 subsequent siblings)
  25 siblings, 1 reply; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

When processing mmap2 event, check on the build id
misc bit: PERF_RECORD_MISC_MMAP_BUILD_ID and if it
is set, store the build id in mmap's dso object.

Also adding the build id data arts to struct
perf_record_mmap2 event definition.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/lib/perf/include/perf/event.h | 18 ++++++++++++++----
 tools/perf/util/machine.c           | 24 +++++++++++++++++++-----
 tools/perf/util/map.c               |  8 ++++++--
 tools/perf/util/map.h               |  3 ++-
 4 files changed, 41 insertions(+), 12 deletions(-)

diff --git a/tools/lib/perf/include/perf/event.h b/tools/lib/perf/include/perf/event.h
index 988c539bedb6..d82054225fcc 100644
--- a/tools/lib/perf/include/perf/event.h
+++ b/tools/lib/perf/include/perf/event.h
@@ -23,10 +23,20 @@ struct perf_record_mmap2 {
 	__u64			 start;
 	__u64			 len;
 	__u64			 pgoff;
-	__u32			 maj;
-	__u32			 min;
-	__u64			 ino;
-	__u64			 ino_generation;
+	union {
+		struct {
+			__u32	 maj;
+			__u32	 min;
+			__u64	 ino;
+			__u64	 ino_generation;
+		};
+		struct {
+			__u8	 build_id_size;
+			__u8	 __reserved_1;
+			__u16	 __reserved_2;
+			__u8	 build_id[20];
+		};
+	};
 	__u32			 prot;
 	__u32			 flags;
 	char			 filename[PATH_MAX];
diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index 1ae32a81639c..1edb7d10b042 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -1599,7 +1599,8 @@ static int machine__process_extra_kernel_map(struct machine *machine,
 }
 
 static int machine__process_kernel_mmap_event(struct machine *machine,
-					      struct extra_kernel_map *xm)
+					      struct extra_kernel_map *xm,
+					      struct build_id *bid)
 {
 	struct map *map;
 	enum dso_space_type dso_space;
@@ -1624,6 +1625,10 @@ static int machine__process_kernel_mmap_event(struct machine *machine,
 			goto out_problem;
 
 		map->end = map->start + xm->end - xm->start;
+
+		if (build_id__is_defined(bid))
+			dso__set_build_id(map->dso, bid);
+
 	} else if (is_kernel_mmap) {
 		const char *symbol_name = (xm->name + strlen(machine->mmap_name));
 		/*
@@ -1681,6 +1686,9 @@ static int machine__process_kernel_mmap_event(struct machine *machine,
 
 		machine__update_kernel_mmap(machine, xm->start, xm->end);
 
+		if (build_id__is_defined(bid))
+			dso__set_build_id(kernel, bid);
+
 		/*
 		 * Avoid using a zero address (kptr_restrict) for the ref reloc
 		 * symbol. Effectively having zero here means that at record
@@ -1718,11 +1726,17 @@ int machine__process_mmap2_event(struct machine *machine,
 		.ino = event->mmap2.ino,
 		.ino_generation = event->mmap2.ino_generation,
 	};
+	struct build_id __bid, *bid = NULL;
 	int ret = 0;
 
 	if (dump_trace)
 		perf_event__fprintf_mmap2(event, stdout);
 
+	if (event->header.misc & PERF_RECORD_MISC_MMAP_BUILD_ID) {
+		bid = &__bid;
+		build_id__init(bid, event->mmap2.build_id, event->mmap2.build_id_size);
+	}
+
 	if (sample->cpumode == PERF_RECORD_MISC_GUEST_KERNEL ||
 	    sample->cpumode == PERF_RECORD_MISC_KERNEL) {
 		struct extra_kernel_map xm = {
@@ -1732,7 +1746,7 @@ int machine__process_mmap2_event(struct machine *machine,
 		};
 
 		strlcpy(xm.name, event->mmap2.filename, KMAP_NAME_LEN);
-		ret = machine__process_kernel_mmap_event(machine, &xm);
+		ret = machine__process_kernel_mmap_event(machine, &xm, bid);
 		if (ret < 0)
 			goto out_problem;
 		return 0;
@@ -1746,7 +1760,7 @@ int machine__process_mmap2_event(struct machine *machine,
 	map = map__new(machine, event->mmap2.start,
 			event->mmap2.len, event->mmap2.pgoff,
 			&dso_id, event->mmap2.prot,
-			event->mmap2.flags,
+			event->mmap2.flags, bid,
 			event->mmap2.filename, thread);
 
 	if (map == NULL)
@@ -1789,7 +1803,7 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 		};
 
 		strlcpy(xm.name, event->mmap.filename, KMAP_NAME_LEN);
-		ret = machine__process_kernel_mmap_event(machine, &xm);
+		ret = machine__process_kernel_mmap_event(machine, &xm, NULL);
 		if (ret < 0)
 			goto out_problem;
 		return 0;
@@ -1805,7 +1819,7 @@ int machine__process_mmap_event(struct machine *machine, union perf_event *event
 
 	map = map__new(machine, event->mmap.start,
 			event->mmap.len, event->mmap.pgoff,
-			NULL, prot, 0, event->mmap.filename, thread);
+			NULL, prot, 0, NULL, event->mmap.filename, thread);
 
 	if (map == NULL)
 		goto out_problem_map;
diff --git a/tools/perf/util/map.c b/tools/perf/util/map.c
index f44ede437dc7..692e56dc832e 100644
--- a/tools/perf/util/map.c
+++ b/tools/perf/util/map.c
@@ -130,8 +130,8 @@ void map__init(struct map *map, u64 start, u64 end, u64 pgoff, struct dso *dso)
 
 struct map *map__new(struct machine *machine, u64 start, u64 len,
 		     u64 pgoff, struct dso_id *id,
-		     u32 prot, u32 flags, char *filename,
-		     struct thread *thread)
+		     u32 prot, u32 flags, struct build_id *bid,
+		     char *filename, struct thread *thread)
 {
 	struct map *map = malloc(sizeof(*map));
 	struct nsinfo *nsi = NULL;
@@ -194,6 +194,10 @@ struct map *map__new(struct machine *machine, u64 start, u64 len,
 				dso__set_loaded(dso);
 		}
 		dso->nsinfo = nsi;
+
+		if (build_id__is_defined(bid))
+			dso__set_build_id(dso, bid);
+
 		dso__put(dso);
 	}
 	return map;
diff --git a/tools/perf/util/map.h b/tools/perf/util/map.h
index b1c0686db1b7..9f32825c98d8 100644
--- a/tools/perf/util/map.h
+++ b/tools/perf/util/map.h
@@ -104,10 +104,11 @@ void map__init(struct map *map,
 	       u64 start, u64 end, u64 pgoff, struct dso *dso);
 
 struct dso_id;
+struct build_id;
 
 struct map *map__new(struct machine *machine, u64 start, u64 len,
 		     u64 pgoff, struct dso_id *id, u32 prot, u32 flags,
-		     char *filename, struct thread *thread);
+		     struct build_id *bid, char *filename, struct thread *thread);
 struct map *map__new2(u64 start, struct dso *dso);
 void map__delete(struct map *map);
 struct map *map__clone(struct map *map);
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 14/25] perf tools: Allow mmap2 event to synthesize kernel image
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (12 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 13/25] perf tools: Store build id from mmap2 events Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 15/25] perf tools: Allow mmap2 event to synthesize modules Jiri Olsa
                   ` (11 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Allow mmap2 event to synthesize kernel image,
so we can synthesize kernel build id data in
following changes.

It's enabled by new symbol_conf.buildid_mmap2
bool, which will be switched in following
changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/symbol_conf.h      |  3 ++-
 tools/perf/util/synthetic-events.c | 40 ++++++++++++++++++++----------
 2 files changed, 29 insertions(+), 14 deletions(-)

diff --git a/tools/perf/util/symbol_conf.h b/tools/perf/util/symbol_conf.h
index b916afb95ec5..b18f9c8dbb75 100644
--- a/tools/perf/util/symbol_conf.h
+++ b/tools/perf/util/symbol_conf.h
@@ -42,7 +42,8 @@ struct symbol_conf {
 			report_block,
 			report_individual_block,
 			inline_name,
-			disable_add2line_warn;
+			disable_add2line_warn,
+			buildid_mmap2;
 	const char	*vmlinux_name,
 			*kallsyms_name,
 			*source_prefix,
diff --git a/tools/perf/util/synthetic-events.c b/tools/perf/util/synthetic-events.c
index 8a23391558cf..872df6d6dbef 100644
--- a/tools/perf/util/synthetic-events.c
+++ b/tools/perf/util/synthetic-events.c
@@ -988,11 +988,12 @@ static int __perf_event__synthesize_kernel_mmap(struct perf_tool *tool,
 						perf_event__handler_t process,
 						struct machine *machine)
 {
-	size_t size;
+	union perf_event *event;
+	size_t size = symbol_conf.buildid_mmap2 ?
+			sizeof(event->mmap2) : sizeof(event->mmap);
 	struct map *map = machine__kernel_map(machine);
 	struct kmap *kmap;
 	int err;
-	union perf_event *event;
 
 	if (map == NULL)
 		return -1;
@@ -1006,7 +1007,7 @@ static int __perf_event__synthesize_kernel_mmap(struct perf_tool *tool,
 	 * available use this, and after it is use this as a fallback for older
 	 * kernels.
 	 */
-	event = zalloc((sizeof(event->mmap) + machine->id_hdr_size));
+	event = zalloc(size + machine->id_hdr_size);
 	if (event == NULL) {
 		pr_debug("Not enough memory synthesizing mmap event "
 			 "for kernel modules\n");
@@ -1023,16 +1024,29 @@ static int __perf_event__synthesize_kernel_mmap(struct perf_tool *tool,
 		event->header.misc = PERF_RECORD_MISC_GUEST_KERNEL;
 	}
 
-	size = snprintf(event->mmap.filename, sizeof(event->mmap.filename),
-			"%s%s", machine->mmap_name, kmap->ref_reloc_sym->name) + 1;
-	size = PERF_ALIGN(size, sizeof(u64));
-	event->mmap.header.type = PERF_RECORD_MMAP;
-	event->mmap.header.size = (sizeof(event->mmap) -
-			(sizeof(event->mmap.filename) - size) + machine->id_hdr_size);
-	event->mmap.pgoff = kmap->ref_reloc_sym->addr;
-	event->mmap.start = map->start;
-	event->mmap.len   = map->end - event->mmap.start;
-	event->mmap.pid   = machine->pid;
+	if (symbol_conf.buildid_mmap2) {
+		size = snprintf(event->mmap2.filename, sizeof(event->mmap2.filename),
+				"%s%s", machine->mmap_name, kmap->ref_reloc_sym->name) + 1;
+		size = PERF_ALIGN(size, sizeof(u64));
+		event->mmap2.header.type = PERF_RECORD_MMAP2;
+		event->mmap2.header.size = (sizeof(event->mmap2) -
+				(sizeof(event->mmap2.filename) - size) + machine->id_hdr_size);
+		event->mmap2.pgoff = kmap->ref_reloc_sym->addr;
+		event->mmap2.start = map->start;
+		event->mmap2.len   = map->end - event->mmap.start;
+		event->mmap2.pid   = machine->pid;
+	} else {
+		size = snprintf(event->mmap.filename, sizeof(event->mmap.filename),
+				"%s%s", machine->mmap_name, kmap->ref_reloc_sym->name) + 1;
+		size = PERF_ALIGN(size, sizeof(u64));
+		event->mmap.header.type = PERF_RECORD_MMAP;
+		event->mmap.header.size = (sizeof(event->mmap) -
+				(sizeof(event->mmap.filename) - size) + machine->id_hdr_size);
+		event->mmap.pgoff = kmap->ref_reloc_sym->addr;
+		event->mmap.start = map->start;
+		event->mmap.len   = map->end - event->mmap.start;
+		event->mmap.pid   = machine->pid;
+	}
 
 	err = perf_tool__process_synth_event(tool, event, machine, process);
 	free(event);
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 15/25] perf tools: Allow mmap2 event to synthesize modules
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (13 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 14/25] perf tools: Allow mmap2 event to synthesize kernel image Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 16/25] perf tools: Synthesize build id for kernel/modules/tasks Jiri Olsa
                   ` (10 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ian Rogers, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Allow mmap2 event to synthesize kernel modules,
so we can synthesize module's build id data in
following changes.

It's enabled by new symbol_conf.buildid_mmap2
bool, which will be switched in following
changes.

Acked-by: Ian Rogers <irogers@google.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/synthetic-events.c | 49 +++++++++++++++++++-----------
 1 file changed, 32 insertions(+), 17 deletions(-)

diff --git a/tools/perf/util/synthetic-events.c b/tools/perf/util/synthetic-events.c
index 872df6d6dbef..a18ae502d765 100644
--- a/tools/perf/util/synthetic-events.c
+++ b/tools/perf/util/synthetic-events.c
@@ -593,16 +593,17 @@ int perf_event__synthesize_modules(struct perf_tool *tool, perf_event__handler_t
 	int rc = 0;
 	struct map *pos;
 	struct maps *maps = machine__kernel_maps(machine);
-	union perf_event *event = zalloc((sizeof(event->mmap) +
-					  machine->id_hdr_size));
+	union perf_event *event;
+	size_t size = symbol_conf.buildid_mmap2 ?
+			sizeof(event->mmap2) : sizeof(event->mmap);
+
+	event = zalloc(size + machine->id_hdr_size);
 	if (event == NULL) {
 		pr_debug("Not enough memory synthesizing mmap event "
 			 "for kernel modules\n");
 		return -1;
 	}
 
-	event->header.type = PERF_RECORD_MMAP;
-
 	/*
 	 * kernel uses 0 for user space maps, see kernel/perf_event.c
 	 * __perf_event_mmap
@@ -613,23 +614,37 @@ int perf_event__synthesize_modules(struct perf_tool *tool, perf_event__handler_t
 		event->header.misc = PERF_RECORD_MISC_GUEST_KERNEL;
 
 	maps__for_each_entry(maps, pos) {
-		size_t size;
-
 		if (!__map__is_kmodule(pos))
 			continue;
 
-		size = PERF_ALIGN(pos->dso->long_name_len + 1, sizeof(u64));
-		event->mmap.header.type = PERF_RECORD_MMAP;
-		event->mmap.header.size = (sizeof(event->mmap) -
-				        (sizeof(event->mmap.filename) - size));
-		memset(event->mmap.filename + size, 0, machine->id_hdr_size);
-		event->mmap.header.size += machine->id_hdr_size;
-		event->mmap.start = pos->start;
-		event->mmap.len   = pos->end - pos->start;
-		event->mmap.pid   = machine->pid;
+		if (symbol_conf.buildid_mmap2) {
+			size = PERF_ALIGN(pos->dso->long_name_len + 1, sizeof(u64));
+			event->mmap2.header.type = PERF_RECORD_MMAP2;
+			event->mmap2.header.size = (sizeof(event->mmap2) -
+						(sizeof(event->mmap2.filename) - size));
+			memset(event->mmap2.filename + size, 0, machine->id_hdr_size);
+			event->mmap2.header.size += machine->id_hdr_size;
+			event->mmap2.start = pos->start;
+			event->mmap2.len   = pos->end - pos->start;
+			event->mmap2.pid   = machine->pid;
+
+			memcpy(event->mmap2.filename, pos->dso->long_name,
+			       pos->dso->long_name_len + 1);
+		} else {
+			size = PERF_ALIGN(pos->dso->long_name_len + 1, sizeof(u64));
+			event->mmap.header.type = PERF_RECORD_MMAP;
+			event->mmap.header.size = (sizeof(event->mmap) -
+						(sizeof(event->mmap.filename) - size));
+			memset(event->mmap.filename + size, 0, machine->id_hdr_size);
+			event->mmap.header.size += machine->id_hdr_size;
+			event->mmap.start = pos->start;
+			event->mmap.len   = pos->end - pos->start;
+			event->mmap.pid   = machine->pid;
+
+			memcpy(event->mmap.filename, pos->dso->long_name,
+			       pos->dso->long_name_len + 1);
+		}
 
-		memcpy(event->mmap.filename, pos->dso->long_name,
-		       pos->dso->long_name_len + 1);
 		if (perf_tool__process_synth_event(tool, event, machine, process) != 0) {
 			rc = -1;
 			break;
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 16/25] perf tools: Synthesize build id for kernel/modules/tasks
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (14 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 15/25] perf tools: Allow mmap2 event to synthesize modules Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 17/25] perf tools: Add support to display build id for mmap2 events Jiri Olsa
                   ` (9 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding build id to synthesized mmap2 events for
everything - kernel/modules/tasks.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/synthetic-events.c | 32 ++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/tools/perf/util/synthetic-events.c b/tools/perf/util/synthetic-events.c
index a18ae502d765..91b1962d399c 100644
--- a/tools/perf/util/synthetic-events.c
+++ b/tools/perf/util/synthetic-events.c
@@ -347,6 +347,31 @@ static bool read_proc_maps_line(struct io *io, __u64 *start, __u64 *end,
 	}
 }
 
+static void perf_record_mmap2__read_build_id(struct perf_record_mmap2 *event,
+					     bool is_kernel)
+{
+	struct build_id bid;
+	int rc;
+
+	if (is_kernel)
+		rc = sysfs__read_build_id("/sys/kernel/notes", &bid);
+	else
+		rc = filename__read_build_id(event->filename, &bid) > 0 ? 0 : -1;
+
+	if (rc == 0) {
+		memcpy(event->build_id, bid.data, sizeof(bid.data));
+		event->build_id_size = (u8) bid.size;
+		event->header.misc |= PERF_RECORD_MISC_MMAP_BUILD_ID;
+		event->__reserved_1 = 0;
+		event->__reserved_2 = 0;
+	} else {
+		if (event->filename[0] == '/') {
+			pr_debug2("Failed to read build ID for %s\n",
+				  event->filename);
+		}
+	}
+}
+
 int perf_event__synthesize_mmap_events(struct perf_tool *tool,
 				       union perf_event *event,
 				       pid_t pid, pid_t tgid,
@@ -453,6 +478,9 @@ int perf_event__synthesize_mmap_events(struct perf_tool *tool,
 		event->mmap2.pid = tgid;
 		event->mmap2.tid = pid;
 
+		if (symbol_conf.buildid_mmap2)
+			perf_record_mmap2__read_build_id(&event->mmap2, false);
+
 		if (perf_tool__process_synth_event(tool, event, machine, process) != 0) {
 			rc = -1;
 			break;
@@ -630,6 +658,8 @@ int perf_event__synthesize_modules(struct perf_tool *tool, perf_event__handler_t
 
 			memcpy(event->mmap2.filename, pos->dso->long_name,
 			       pos->dso->long_name_len + 1);
+
+			perf_record_mmap2__read_build_id(&event->mmap2, false);
 		} else {
 			size = PERF_ALIGN(pos->dso->long_name_len + 1, sizeof(u64));
 			event->mmap.header.type = PERF_RECORD_MMAP;
@@ -1050,6 +1080,8 @@ static int __perf_event__synthesize_kernel_mmap(struct perf_tool *tool,
 		event->mmap2.start = map->start;
 		event->mmap2.len   = map->end - event->mmap.start;
 		event->mmap2.pid   = machine->pid;
+
+		perf_record_mmap2__read_build_id(&event->mmap2, true);
 	} else {
 		size = snprintf(event->mmap.filename, sizeof(event->mmap.filename),
 				"%s%s", machine->mmap_name, kmap->ref_reloc_sym->name) + 1;
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 17/25] perf tools: Add support to display build id for mmap2 events
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (15 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 16/25] perf tools: Synthesize build id for kernel/modules/tasks Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids Jiri Olsa
                   ` (8 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ian Rogers, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding support to display build id in mmap2 events:

  $ perf script --show-mmap-events | head -4
  swapper ... @ 0xffffffff81000000 <ff1969b3ba5e43911208bb46fa7d5b1eb809e422>]: ---p [kernel.kallsyms]_text
  swapper ... @ 0 <5f62adb730272c9417883ae8b8a8ec224df8cddd>]: ---p /lib/modules/5.9.0-rc5buildid+/kernel/drivers/firmware/qemu_fw_cfg.ko
  swapper ... @ 0 <c9ac6e1dafc1ebdadb048f967854e810706c8bab>]: ---p /lib/modules/5.9.0-rc5buildid+/kernel/drivers/char/virtio_console.ko
  swapper ... @ 0 <86441a4c5b2c2ff5b440682f4c612bd4b426eb5c>]: ---p /lib/modules/5.9.0-rc5buildid+/kernel/lib/libcrc32c.ko

  $ perf report -D | grep MMAP2 | head -4
  0 0 ... @ 0xffffffff81000000 <ff1969b3ba5e43911208bb46fa7d5b1eb809e422>]: ---p [kernel.kallsyms]_text
  0 0 ... @ 0 <5f62adb730272c9417883ae8b8a8ec224df8cddd>]: ---p /lib/modules/5.9.0-rc5buildid+/kernel/drivers/firmware/qemu_fw_cfg.ko
  0 0 ... @ 0 <c9ac6e1dafc1ebdadb048f967854e810706c8bab>]: ---p /lib/modules/5.9.0-rc5buildid+/kernel/drivers/char/virtio_console.ko
  0 0 ... @ 0 <86441a4c5b2c2ff5b440682f4c612bd4b426eb5c>]: ---p /lib/modules/5.9.0-rc5buildid+/kernel/lib/libcrc32c.ko

Adding build id data into <> brackets.

Acked-by: Ian Rogers <irogers@google.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/event.c | 41 ++++++++++++++++++++++++++++++-----------
 1 file changed, 30 insertions(+), 11 deletions(-)

diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 05616d4138a9..fbe8578e4c47 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -288,17 +288,36 @@ size_t perf_event__fprintf_mmap(union perf_event *event, FILE *fp)
 
 size_t perf_event__fprintf_mmap2(union perf_event *event, FILE *fp)
 {
-	return fprintf(fp, " %d/%d: [%#" PRI_lx64 "(%#" PRI_lx64 ") @ %#" PRI_lx64
-			   " %02x:%02x %"PRI_lu64" %"PRI_lu64"]: %c%c%c%c %s\n",
-		       event->mmap2.pid, event->mmap2.tid, event->mmap2.start,
-		       event->mmap2.len, event->mmap2.pgoff, event->mmap2.maj,
-		       event->mmap2.min, event->mmap2.ino,
-		       event->mmap2.ino_generation,
-		       (event->mmap2.prot & PROT_READ) ? 'r' : '-',
-		       (event->mmap2.prot & PROT_WRITE) ? 'w' : '-',
-		       (event->mmap2.prot & PROT_EXEC) ? 'x' : '-',
-		       (event->mmap2.flags & MAP_SHARED) ? 's' : 'p',
-		       event->mmap2.filename);
+	if (event->header.misc & PERF_RECORD_MISC_MMAP_BUILD_ID) {
+		char sbuild_id[SBUILD_ID_SIZE];
+		struct build_id bid;
+
+		build_id__init(&bid, event->mmap2.build_id,
+			       event->mmap2.build_id_size);
+		build_id__sprintf(&bid, sbuild_id);
+
+		return fprintf(fp, " %d/%d: [%#" PRI_lx64 "(%#" PRI_lx64 ") @ %#" PRI_lx64
+				   " <%s>]: %c%c%c%c %s\n",
+			       event->mmap2.pid, event->mmap2.tid, event->mmap2.start,
+			       event->mmap2.len, event->mmap2.pgoff, sbuild_id,
+			       (event->mmap2.prot & PROT_READ) ? 'r' : '-',
+			       (event->mmap2.prot & PROT_WRITE) ? 'w' : '-',
+			       (event->mmap2.prot & PROT_EXEC) ? 'x' : '-',
+			       (event->mmap2.flags & MAP_SHARED) ? 's' : 'p',
+			       event->mmap2.filename);
+	} else {
+		return fprintf(fp, " %d/%d: [%#" PRI_lx64 "(%#" PRI_lx64 ") @ %#" PRI_lx64
+				   " %02x:%02x %"PRI_lu64" %"PRI_lu64"]: %c%c%c%c %s\n",
+			       event->mmap2.pid, event->mmap2.tid, event->mmap2.start,
+			       event->mmap2.len, event->mmap2.pgoff, event->mmap2.maj,
+			       event->mmap2.min, event->mmap2.ino,
+			       event->mmap2.ino_generation,
+			       (event->mmap2.prot & PROT_READ) ? 'r' : '-',
+			       (event->mmap2.prot & PROT_WRITE) ? 'w' : '-',
+			       (event->mmap2.prot & PROT_EXEC) ? 'x' : '-',
+			       (event->mmap2.flags & MAP_SHARED) ? 's' : 'p',
+			       event->mmap2.filename);
+	}
 }
 
 size_t perf_event__fprintf_thread_map(union perf_event *event, FILE *fp)
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (16 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 17/25] perf tools: Add support to display build id for mmap2 events Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 19/25] perf tools: Add __perf_session__cache_build_ids function Jiri Olsa
                   ` (7 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Using machine__for_each_dso in perf_session__cache_build_ids,
so we can reuse perf_session__cache_build_ids with different
callback in following changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 41 +++++++++++++++-----------------------
 1 file changed, 16 insertions(+), 25 deletions(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 1fd58703c2d4..948a7f48d668 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -859,12 +859,16 @@ int build_id_cache__remove_s(const char *sbuild_id)
 	return err;
 }
 
-static int dso__cache_build_id(struct dso *dso, struct machine *machine)
+static int dso__cache_build_id(struct dso *dso, struct machine *machine,
+			       void *priv __maybe_unused)
 {
 	bool is_kallsyms = dso__is_kallsyms(dso);
 	bool is_vdso = dso__is_vdso(dso);
 	const char *name = dso->long_name;
 
+	if (!dso->has_build_id)
+		return 0;
+
 	if (dso__is_kcore(dso)) {
 		is_kallsyms = true;
 		name = machine->mmap_name;
@@ -873,43 +877,30 @@ static int dso__cache_build_id(struct dso *dso, struct machine *machine)
 				     is_kallsyms, is_vdso);
 }
 
-static int __dsos__cache_build_ids(struct list_head *head,
-				   struct machine *machine)
+static int
+machines__for_each_dso(struct machines *machines, machine__dso_t fn, void *priv)
 {
-	struct dso *pos;
-	int err = 0;
-
-	dsos__for_each_with_build_id(pos, head)
-		if (dso__cache_build_id(pos, machine))
-			err = -1;
+	int ret = machine__for_each_dso(&machines->host, fn, priv);
+	struct rb_node *nd;
 
-	return err;
-}
+	for (nd = rb_first_cached(&machines->guests); nd;
+	     nd = rb_next(nd)) {
+		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 
-static int machine__cache_build_ids(struct machine *machine)
-{
-	return __dsos__cache_build_ids(&machine->dsos.head, machine);
+		ret |= machine__for_each_dso(pos, fn, priv);
+	}
+	return ret ? -1 : 0;
 }
 
 int perf_session__cache_build_ids(struct perf_session *session)
 {
-	struct rb_node *nd;
-	int ret;
-
 	if (no_buildid_cache)
 		return 0;
 
 	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
 		return -1;
 
-	ret = machine__cache_build_ids(&session->machines.host);
-
-	for (nd = rb_first_cached(&session->machines.guests); nd;
-	     nd = rb_next(nd)) {
-		struct machine *pos = rb_entry(nd, struct machine, rb_node);
-		ret |= machine__cache_build_ids(pos);
-	}
-	return ret ? -1 : 0;
+	return machines__for_each_dso(&session->machines, dso__cache_build_id, NULL) ?  -1 : 0;
 }
 
 static bool machine__read_build_ids(struct machine *machine, bool with_hits)
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 19/25] perf tools: Add __perf_session__cache_build_ids function
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (17 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 20/25] perf tools: Add is_perf_data function Jiri Olsa
                   ` (6 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding __perf_session__cache_build_ids function as an
interface for caching sessions build ids with callback
function and its data pointer.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 10 ++++++++--
 tools/perf/util/build-id.h |  3 +++
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 948a7f48d668..6bf3cc79c5d5 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -892,7 +892,8 @@ machines__for_each_dso(struct machines *machines, machine__dso_t fn, void *priv)
 	return ret ? -1 : 0;
 }
 
-int perf_session__cache_build_ids(struct perf_session *session)
+int __perf_session__cache_build_ids(struct perf_session *session,
+				    machine__dso_t fn, void *priv)
 {
 	if (no_buildid_cache)
 		return 0;
@@ -900,7 +901,12 @@ int perf_session__cache_build_ids(struct perf_session *session)
 	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
 		return -1;
 
-	return machines__for_each_dso(&session->machines, dso__cache_build_id, NULL) ?  -1 : 0;
+	return machines__for_each_dso(&session->machines, fn, priv) ?  -1 : 0;
+}
+
+int perf_session__cache_build_ids(struct perf_session *session)
+{
+	return __perf_session__cache_build_ids(session, dso__cache_build_id, NULL);
 }
 
 static bool machine__read_build_ids(struct machine *machine, bool with_hits)
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index f1a2f67df6e4..c6f10e3d2152 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -5,6 +5,7 @@
 #define BUILD_ID_SIZE	20
 #define SBUILD_ID_SIZE	(BUILD_ID_SIZE * 2 + 1)
 
+#include "machine.h"
 #include "tool.h"
 #include <linux/types.h>
 
@@ -46,6 +47,8 @@ bool perf_session__read_build_ids(struct perf_session *session, bool with_hits);
 int perf_session__write_buildid_table(struct perf_session *session,
 				      struct feat_fd *fd);
 int perf_session__cache_build_ids(struct perf_session *session);
+int __perf_session__cache_build_ids(struct perf_session *session,
+				    machine__dso_t fn, void *priv);
 
 char *build_id_cache__origname(const char *sbuild_id);
 char *build_id_cache__linkname(const char *sbuild_id, char *bf, size_t size);
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 20/25] perf tools: Add is_perf_data function
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (18 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 19/25] perf tools: Add __perf_session__cache_build_ids function Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 21/25] perf tools: Add build_id_cache__add function Jiri Olsa
                   ` (5 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ian Rogers, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding is_perf_data function that returns true if
the given path is perf data file. It will be used
in following patches.

Acked-by: Ian Rogers <irogers@google.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/data.c | 19 +++++++++++++++++++
 tools/perf/util/data.h |  1 +
 2 files changed, 20 insertions(+)

diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c
index 05bbcb663c41..f29af4fc3d09 100644
--- a/tools/perf/util/data.c
+++ b/tools/perf/util/data.c
@@ -492,3 +492,22 @@ char *perf_data__kallsyms_name(struct perf_data *data)
 
 	return kallsyms_name;
 }
+
+bool is_perf_data(const char *path)
+{
+	bool ret = false;
+	FILE *file;
+	u64 magic;
+
+	file = fopen(path, "r");
+	if (!file)
+		return false;
+
+	if (fread(&magic, 1, 8, file) < 8)
+		goto out;
+
+	ret = is_perf_magic(magic);
+out:
+	fclose(file);
+	return ret;
+}
diff --git a/tools/perf/util/data.h b/tools/perf/util/data.h
index c563fcbb0288..62a3e66fbee8 100644
--- a/tools/perf/util/data.h
+++ b/tools/perf/util/data.h
@@ -98,4 +98,5 @@ int perf_data__update_dir(struct perf_data *data);
 unsigned long perf_data__size(struct perf_data *data);
 int perf_data__make_kcore_dir(struct perf_data *data, char *buf, size_t buf_sz);
 char *perf_data__kallsyms_name(struct perf_data *data);
+bool is_perf_data(const char *path);
 #endif /* __PERF_DATA_H */
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 21/25] perf tools: Add build_id_cache__add function
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (19 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 20/25] perf tools: Add is_perf_data function Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data Jiri Olsa
                   ` (4 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ian Rogers, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding build_id_cache__add function as core function
that adds file into build id database. It will be
set from another callers in following changes.

Acked-by: Ian Rogers <irogers@google.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 42 ++++++++++++++++++++++++--------------
 tools/perf/util/build-id.h |  2 ++
 2 files changed, 29 insertions(+), 15 deletions(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 6bf3cc79c5d5..02df36b30ac5 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -671,24 +671,15 @@ static char *build_id_cache__find_debug(const char *sbuild_id,
 	return realname;
 }
 
-int build_id_cache__add_s(const char *sbuild_id, const char *name,
-			  struct nsinfo *nsi, bool is_kallsyms, bool is_vdso)
+int
+build_id_cache__add(const char *sbuild_id, const char *name, const char *realname,
+		    struct nsinfo *nsi, bool is_kallsyms, bool is_vdso)
 {
 	const size_t size = PATH_MAX;
-	char *realname = NULL, *filename = NULL, *dir_name = NULL,
-	     *linkname = zalloc(size), *tmp;
+	char *filename = NULL, *dir_name = NULL, *linkname = zalloc(size), *tmp;
 	char *debugfile = NULL;
 	int err = -1;
 
-	if (!is_kallsyms) {
-		if (!is_vdso)
-			realname = nsinfo__realpath(name, nsi);
-		else
-			realname = realpath(name, NULL);
-		if (!realname)
-			goto out_free;
-	}
-
 	dir_name = build_id_cache__cachedir(sbuild_id, name, nsi, is_kallsyms,
 					    is_vdso);
 	if (!dir_name)
@@ -788,8 +779,6 @@ int build_id_cache__add_s(const char *sbuild_id, const char *name,
 		pr_debug4("Failed to update/scan SDT cache for %s\n", realname);
 
 out_free:
-	if (!is_kallsyms)
-		free(realname);
 	free(filename);
 	free(debugfile);
 	free(dir_name);
@@ -797,6 +786,29 @@ int build_id_cache__add_s(const char *sbuild_id, const char *name,
 	return err;
 }
 
+int build_id_cache__add_s(const char *sbuild_id, const char *name,
+			  struct nsinfo *nsi, bool is_kallsyms, bool is_vdso)
+{
+	char *realname = NULL;
+	int err = -1;
+
+	if (!is_kallsyms) {
+		if (!is_vdso)
+			realname = nsinfo__realpath(name, nsi);
+		else
+			realname = realpath(name, NULL);
+		if (!realname)
+			goto out_free;
+	}
+
+	err = build_id_cache__add(sbuild_id, name, realname, nsi, is_kallsyms, is_vdso);
+
+out_free:
+	if (!is_kallsyms)
+		free(realname);
+	return err;
+}
+
 static int build_id_cache__add_b(const struct build_id *bid,
 				 const char *name, struct nsinfo *nsi,
 				 bool is_kallsyms, bool is_vdso)
diff --git a/tools/perf/util/build-id.h b/tools/perf/util/build-id.h
index c6f10e3d2152..02613f4b2c29 100644
--- a/tools/perf/util/build-id.h
+++ b/tools/perf/util/build-id.h
@@ -63,6 +63,8 @@ char *build_id_cache__complement(const char *incomplete_sbuild_id);
 int build_id_cache__list_build_ids(const char *pathname, struct nsinfo *nsi,
 				   struct strlist **result);
 bool build_id_cache__cached(const char *sbuild_id);
+int build_id_cache__add(const char *sbuild_id, const char *name, const char *realname,
+			struct nsinfo *nsi, bool is_kallsyms, bool is_vdso);
 int build_id_cache__add_s(const char *sbuild_id,
 			  const char *name, struct nsinfo *nsi,
 			  bool is_kallsyms, bool is_vdso);
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (20 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 21/25] perf tools: Add build_id_cache__add function Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-25 13:00   ` Namhyung Kim
  2020-11-25 13:29   ` Namhyung Kim
  2020-11-23 23:05 ` [PATCH 23/25] perf buildid-cache: Add --debuginfod option Jiri Olsa
                   ` (3 subsequent siblings)
  25 siblings, 2 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding support to specify perf data file as -a option file
argument,

If the file is detected to be perf data file, it is processed
and all dso objects with sample hit are stored to the build
id cache.

  $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data
  OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
  OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko

By default we store only dso with hits, but it's possible to
specify 'all' to store all dso objects, like:
    -a perf.data,all

  $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data,all
  OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
  OK   6ce92dc7c31f12fe5b7775a2bb8b14a3546ce2cd /lib/modules/5.10.0-rc2speed+/kernel/drivers/firmware/qemu_fw_cfg.ko
  OK   bf3f6d32dccc159f841fc3658c241d0e74c61fbb /lib/modules/5.10.0-rc2speed+/kernel/drivers/block/virtio_blk.ko
  OK   e896b4329cf9f190f1a0fae933f425ff8f71b052 /lib/modules/5.10.0-rc2speed+/kernel/drivers/char/virtio_console.ko
  OK   5bedc933cb59e053ecb472f327bd73c548364479 /lib/modules/5.10.0-rc2speed+/kernel/drivers/input/serio/serio_raw.ko
  OK   cecc506368a8b7a473a5f900d26f0d3d914a9c23 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32c-intel.ko
  OK   91076fb3646d061a0a42cf7bddb339a665ee4f80 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/ghash-clmulni-intel.ko
  OK   4e2a304d788bb8e2e950bc82a5944e042afa0bf2 /lib/modules/5.10.0-rc2speed+/kernel/drivers/media/cec/core/cec.ko
  OK   31ab0da5ad81e6803280177f507a95f3053d585e /lib/modules/5.10.0-rc2speed+/kernel/lib/libcrc32c.ko
  OK   f6154bca47c149f48c942fcc3d653041dd285c65 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/ttm/ttm.ko
  OK   723f5852de81590d54b23b38c160d3618b41951b /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crct10dif-pclmul.ko
  OK   06b1eab7f141cbc3e5a5db47909c8ab5cb242e40 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_ttm_helper.ko
  OK   38292b862cf3ff87489508fdb4895efa45780813 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/qxl/qxl.ko
  OK   cdf51e58609bf2ce4837a7b195e0ccae0a930907 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32-pclmul.ko
  OK   5ca8958388f6688452ecc2cb83d6031394c659ad /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm.ko
  OK   236bc4e4f38bf3559007566cb32b3dcc1bc28d2d /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_kms_helper.ko
  OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
  OK   66db2be3efaa43bb5a5c481986e9554e1885cc69 /usr/lib/systemd/systemd
  OK   7db607d9f2de89860d9639712da64c8bacd31e4b /usr/lib64/libm-2.30.so
  OK   55b5f9652e1d17c1dd58f62628d5063428e5db91 /usr/lib64/libudev.so.1.6.15
  OK   63b97070bf097130713bb6c89cf7100b5f3c9b17 /usr/lib64/libunistring.so.2.1.0
  ...

Once perf data is specified, no other file can be specified in
the option, otherwise it causes syntax error.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../perf/Documentation/perf-buildid-cache.txt |  12 +-
 tools/perf/builtin-buildid-cache.c            | 215 +++++++++++++++++-
 tools/perf/util/probe-event.c                 |   6 +-
 3 files changed, 227 insertions(+), 6 deletions(-)

diff --git a/tools/perf/Documentation/perf-buildid-cache.txt b/tools/perf/Documentation/perf-buildid-cache.txt
index f6de0952ff3c..b77da5138bca 100644
--- a/tools/perf/Documentation/perf-buildid-cache.txt
+++ b/tools/perf/Documentation/perf-buildid-cache.txt
@@ -23,7 +23,17 @@ OPTIONS
 -------
 -a::
 --add=::
-        Add specified file to the cache.
+        Add specified file or perf.data binaries to the cache.
+
+        If the file is detected to be perf data file, it is processed
+        and all dso objects with sample hit are stored to the cache.
+
+        It's possible to specify 'all' to store all dso objects, like:
+            -a perf.data,all
+
+        Once perf data is specified, no other file can be specified in
+        the option, otherwise it causes syntax error.
+
 -f::
 --force::
 	Don't complain, do it.
diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index a25411926e48..0bfb54ee1e5e 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -29,6 +29,11 @@
 #include "util/probe-file.h"
 #include <linux/string.h>
 #include <linux/err.h>
+#include <linux/zalloc.h>
+#include <sys/stat.h>
+#ifdef HAVE_DEBUGINFOD_SUPPORT
+#include <elfutils/debuginfod.h>
+#endif
 
 static int build_id_cache__kcore_buildid(const char *proc_dir, char *sbuildid)
 {
@@ -348,6 +353,205 @@ static int build_id_cache__show_all(void)
 	return 0;
 }
 
+#ifdef HAVE_DEBUGINFOD_SUPPORT
+static int call_debuginfod(const char *sbuild_id, char **path, bool debuginfo)
+{
+	debuginfod_client *c;
+	int fd;
+
+	c = debuginfod_begin();
+	if (c == NULL)
+		return -1;
+
+	pr_debug("trying debuginfod for executable <%s> ... ", sbuild_id);
+
+	if (debuginfo) {
+		fd = debuginfod_find_debuginfo(c, (const unsigned char *) sbuild_id,
+					       0, path);
+	} else {
+		fd = debuginfod_find_executable(c, (const unsigned char *) sbuild_id,
+						0, path);
+	}
+	if (fd >= 0)
+		close(fd); /* retaining reference by realname */
+
+	debuginfod_end(c);
+	pr_debug("%s%s\n", *path ? "OK " : "FAILED", *path ? *path : "");
+	return *path ? 0 : -1;
+}
+#else
+static int call_debuginfod(const char *sbuild_id __maybe_unused,
+			   char **path __maybe_unused,
+			   bool debuginfo __maybe_unused)
+{
+	return -1;
+}
+#endif
+
+struct dso_store_data {
+	bool	 hits;
+	bool	 force_download;
+};
+
+static int dso_store(struct dso *dso, struct machine *machine, void *priv)
+{
+	struct dso_store_data *data = priv;
+	char sbuild_id[SBUILD_ID_SIZE];
+	struct build_id bid;
+	char *path = NULL, *link = NULL;
+	bool is_kallsyms;
+	int err = -1;
+
+	/*
+	 * There's no build id in dso, nothing to do..
+	 */
+	if (!dso->has_build_id || !build_id__is_defined(&dso->bid))
+		return 0;
+
+	if (data->hits && !dso->hit)
+		return 0;
+
+	/*
+	 * The storing process is:
+	 *   - get build id of the dso
+	 *   - check if it is already in cache
+	 *   - check if it matches provided build id from mmap2 event
+	 *   - if not, try debuginfod to download the binary
+	 *   - store binary to build id database
+	 */
+	is_kallsyms = !strcmp(machine->mmap_name, dso->short_name);
+	build_id__sprintf(&dso->bid, sbuild_id);
+
+	link = build_id_cache__linkname(sbuild_id, NULL, 0);
+	if (!link)
+		return -ENOMEM;
+
+	if (!data->force_download && !access(link, X_OK)) {
+		pr_debug("already in cache - %s <%s>\n", dso->long_name, sbuild_id);
+		err = 0;
+		goto out;
+	}
+
+	path = strdup(dso->long_name);
+	if (!path)
+		goto out;
+
+	if (is_kallsyms) {
+		/*
+		 * Find out if we are on the same kernel as perf.data
+		 * and store kallsyms in that case.
+		 */
+		err = sysfs__read_build_id("/sys/kernel/notes", &bid);
+		if (err < 0)
+			goto out;
+	} else {
+		struct nscookie nsc;
+		struct stat st;
+
+		nsinfo__mountns_enter(dso->nsinfo, &nsc);
+
+		/*
+		 * Does the file exists in the first place, if it does,
+		 * resolve path and read the build id.
+		 */
+		if (stat(dso->long_name, &st)) {
+			nsinfo__mountns_exit(&nsc);
+			zfree(&path);
+			goto try_download;
+		}
+
+		err = filename__read_build_id(dso->long_name, &bid);
+		nsinfo__mountns_exit(&nsc);
+
+		if (err <= 0)
+			goto out;
+	}
+
+	/*
+	 * If we match, then what we want in mmap2 event
+	 * is what we got in the binary,
+	 */
+	if (bid.size != dso->bid.size || memcmp(&bid, &dso->bid, bid.size)) {
+		char sbid[SBUILD_ID_SIZE];
+
+		build_id__sprintf(&bid, sbid);
+		pr_debug("mmap build id <%s> does not match for %s <%s>\n",
+			 sbuild_id, path, sbid);
+		zfree(&path);
+	}
+
+try_download:
+	/*
+	 * We did not match build id or did not find the
+	 * binary - try debuginfod as last resort.
+	 */
+	if (!path) {
+		bool debuginfo;
+		char *tmp = NULL;
+
+		/*
+		 * The debuginfo retrieval for standard binaries
+		 * is handled within build_id_cache__add function.
+		 *
+		 * For kernel and kernel modules we have to ask
+		 * for debuginfo directly, because debuginfod
+		 * does not treat them as binaries.
+		 */
+		debuginfo = is_kallsyms ||
+			    is_kernel_module(dso->long_name, PERF_RECORD_MISC_CPUMODE_UNKNOWN);
+
+		if (call_debuginfod(sbuild_id, &tmp, debuginfo)) {
+			err = -1;
+			goto out;
+		}
+
+		path = tmp;
+
+		/*
+		 * The kernel dso is now elf binary, so disable is_kallsyms
+		 * so build_id_cache__add can prepare proper file names.
+		 */
+		is_kallsyms = false;
+	}
+
+	pr_debug("linking %s %s <%s>\n", dso->short_name, path, sbuild_id);
+
+	err = build_id_cache__add(sbuild_id, path, path,
+				  dso->nsinfo, is_kallsyms, false);
+out:
+	free(path);
+	fprintf(stderr, "%s %s %s\n", err ? "FAIL" : "OK  ", sbuild_id, dso->long_name);
+	return 0;
+}
+
+static int
+build_id_cache__add_perf_data(const char *path, bool all)
+{
+	struct perf_session *session;
+	struct dso_store_data priv = {
+		.hits = !all,
+		.force_download = false,
+	};
+	struct perf_data data = {
+		.path  = path,
+		.mode  = PERF_DATA_MODE_READ,
+	};
+	int err;
+
+	session = perf_session__new(&data, false, &build_id__mark_dso_hit_ops);
+	if (IS_ERR(session))
+		return PTR_ERR(session);
+
+	err = perf_session__process_events(session);
+	if (err)
+		goto out;
+
+	err = __perf_session__cache_build_ids(session, dso_store, &priv);
+out:
+	perf_session__delete(session);
+	return err;
+}
+
 int cmd_buildid_cache(int argc, const char **argv)
 {
 	struct strlist *list;
@@ -440,7 +644,15 @@ int cmd_buildid_cache(int argc, const char **argv)
 		list = strlist__new(add_name_list_str, NULL);
 		if (list) {
 			strlist__for_each_entry(pos, list)
-				if (build_id_cache__add_file(pos->s, nsi)) {
+				if (is_perf_data(pos->s)) {
+					struct str_node *all_pos = strlist__next(pos);
+					bool all = !strcmp("all", all_pos ? all_pos->s : "");
+
+					if (build_id_cache__add_perf_data(pos->s, all))
+						pr_warning("Couldn't add build ids from %s\n", pos->s);
+					if (all)
+						pos = all_pos;
+				} else if (build_id_cache__add_file(pos->s, nsi)) {
 					if (errno == EEXIST) {
 						pr_debug("%s already in the cache\n",
 							 pos->s);
@@ -449,7 +661,6 @@ int cmd_buildid_cache(int argc, const char **argv)
 					pr_warning("Couldn't add %s: %s\n",
 						   pos->s, str_error_r(errno, sbuf, sizeof(sbuf)));
 				}
-
 			strlist__delete(list);
 		}
 	}
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 8eae2afff71a..e821bb977c9b 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -1616,9 +1616,9 @@ static int parse_perf_probe_point(char *arg, struct perf_probe_event *pev)
 		return -EINVAL;
 	}
 
-	pr_debug("symbol:%s file:%s line:%d offset:%lu return:%d lazy:%s\n",
-		 pp->function, pp->file, pp->line, pp->offset, pp->retprobe,
-		 pp->lazy_line);
+	pr_debug2("symbol:%s file:%s line:%d offset:%lu return:%d lazy:%s\n",
+		  pp->function, pp->file, pp->line, pp->offset, pp->retprobe,
+		  pp->lazy_line);
 	return 0;
 }
 
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 23/25] perf buildid-cache: Add --debuginfod option
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (21 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 24/25] perf buildid-list: Add support for mmap2's buildid events Jiri Olsa
                   ` (2 subsequent siblings)
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ian Rogers, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding --debuginfod option to specify debuginfod url and
support to do that through config file as well.

Use following in ~/.perfconfig file:

  [buildid-cache]
  debuginfod=http://192.168.122.174:8002

Acked-by: Ian Rogers <irogers@google.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 .../perf/Documentation/perf-buildid-cache.txt |  6 ++++
 tools/perf/Documentation/perf-config.txt      |  7 +++++
 tools/perf/builtin-buildid-cache.c            | 28 +++++++++++++++++--
 3 files changed, 38 insertions(+), 3 deletions(-)

diff --git a/tools/perf/Documentation/perf-buildid-cache.txt b/tools/perf/Documentation/perf-buildid-cache.txt
index b77da5138bca..b9987d1399ca 100644
--- a/tools/perf/Documentation/perf-buildid-cache.txt
+++ b/tools/perf/Documentation/perf-buildid-cache.txt
@@ -84,6 +84,12 @@ OPTIONS
 	used when creating a uprobe for a process that resides in a
 	different mount namespace from the perf(1) utility.
 
+--debuginfod=URLs::
+	Specify debuginfod URL to be used when retrieving perf.data binaries,
+	it follows the same syntax as the DEBUGINFOD_URLS variable, like:
+
+	  buildid-cache.debuginfod=http://192.168.122.174:8002
+
 SEE ALSO
 --------
 linkperf:perf-record[1], linkperf:perf-report[1], linkperf:perf-buildid-list[1]
diff --git a/tools/perf/Documentation/perf-config.txt b/tools/perf/Documentation/perf-config.txt
index 31069d8a5304..e3672c5d801b 100644
--- a/tools/perf/Documentation/perf-config.txt
+++ b/tools/perf/Documentation/perf-config.txt
@@ -238,6 +238,13 @@ buildid.*::
 		cache location, or to disable it altogether. If you want to disable it,
 		set buildid.dir to /dev/null. The default is $HOME/.debug
 
+buildid-cache.*::
+	buildid-cache.debuginfod=URLs
+		Specify debuginfod URLs to be used when retrieving perf.data binaries,
+		it follows the same syntax as the DEBUGINFOD_URLS variable, like:
+
+		  buildid-cache.debuginfod=http://192.168.122.174:8002
+
 annotate.*::
 	These are in control of addresses, jump function, source code
 	in lines of assembly code from a specific program.
diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
index 0bfb54ee1e5e..fc03de7d2a28 100644
--- a/tools/perf/builtin-buildid-cache.c
+++ b/tools/perf/builtin-buildid-cache.c
@@ -27,6 +27,7 @@
 #include "util/time-utils.h"
 #include "util/util.h"
 #include "util/probe-file.h"
+#include "util/config.h"
 #include <linux/string.h>
 #include <linux/err.h>
 #include <linux/zalloc.h>
@@ -552,12 +553,21 @@ build_id_cache__add_perf_data(const char *path, bool all)
 	return err;
 }
 
+static int perf_buildid_cache_config(const char *var, const char *value, void *cb)
+{
+	const char **debuginfod = cb;
+
+	if (!strcmp(var, "buildid-cache.debuginfod"))
+		*debuginfod = strdup(value);
+
+	return 0;
+}
+
 int cmd_buildid_cache(int argc, const char **argv)
 {
 	struct strlist *list;
 	struct str_node *pos;
-	int ret = 0;
-	int ns_id = -1;
+	int ret, ns_id = -1;
 	bool force = false;
 	bool list_files = false;
 	bool opts_flag = false;
@@ -567,7 +577,8 @@ int cmd_buildid_cache(int argc, const char **argv)
 		   *purge_name_list_str = NULL,
 		   *missing_filename = NULL,
 		   *update_name_list_str = NULL,
-		   *kcore_filename = NULL;
+		   *kcore_filename = NULL,
+		   *debuginfod = NULL;
 	char sbuf[STRERR_BUFSIZE];
 
 	struct perf_data data = {
@@ -592,6 +603,8 @@ int cmd_buildid_cache(int argc, const char **argv)
 	OPT_BOOLEAN('f', "force", &force, "don't complain, do it"),
 	OPT_STRING('u', "update", &update_name_list_str, "file list",
 		    "file(s) to update"),
+	OPT_STRING(0, "debuginfod", &debuginfod, "debuginfod url",
+		    "set debuginfod url"),
 	OPT_INCR('v', "verbose", &verbose, "be more verbose"),
 	OPT_INTEGER(0, "target-ns", &ns_id, "target pid for namespace context"),
 	OPT_END()
@@ -601,6 +614,10 @@ int cmd_buildid_cache(int argc, const char **argv)
 		NULL
 	};
 
+	ret = perf_config(perf_buildid_cache_config, &debuginfod);
+	if (ret)
+		return ret;
+
 	argc = parse_options(argc, argv, buildid_cache_options,
 			     buildid_cache_usage, 0);
 
@@ -612,6 +629,11 @@ int cmd_buildid_cache(int argc, const char **argv)
 	if (argc || !(list_files || opts_flag))
 		usage_with_options(buildid_cache_usage, buildid_cache_options);
 
+	if (debuginfod) {
+		pr_debug("DEBUGINFOD_URLS=%s\n", debuginfod);
+		setenv("DEBUGINFOD_URLS", debuginfod, 1);
+	}
+
 	/* -l is exclusive. It can not be used with other options. */
 	if (list_files && opts_flag) {
 		usage_with_options_msg(buildid_cache_usage,
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 24/25] perf buildid-list: Add support for mmap2's buildid events
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (22 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 23/25] perf buildid-cache: Add --debuginfod option Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-23 23:05 ` [PATCH 25/25] perf record: Add --buildid-mmap option to enable mmap's build id Jiri Olsa
  2020-11-25 12:54 ` [PATCHv3 00/25] perf: Add mmap2 build id support Namhyung Kim
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Add buildid-list support for mmap2's build id data, so we can
display build ids for dso objects for data without the build
id cache update.

  $ perf buildid-list
  1805c738c8f3ec0f47b7ea09080c28f34d18a82b /usr/lib64/ld-2.31.so
  d278249792061c6b74d1693ca59513be1def13f2 /usr/lib64/libc-2.31.so

By default only dso objects with hits are shown.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/builtin-buildid-list.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/perf/builtin-buildid-list.c b/tools/perf/builtin-buildid-list.c
index e3ef75583514..87f5b1a4a7fa 100644
--- a/tools/perf/builtin-buildid-list.c
+++ b/tools/perf/builtin-buildid-list.c
@@ -77,6 +77,9 @@ static int perf_session__list_build_ids(bool force, bool with_hits)
 	    perf_header__has_feat(&session->header, HEADER_AUXTRACE))
 		with_hits = false;
 
+	if (!perf_header__has_feat(&session->header, HEADER_BUILD_ID))
+		with_hits = true;
+
 	/*
 	 * in pipe-mode, the only way to get the buildids is to parse
 	 * the record stream. Buildids are stored as RECORD_HEADER_BUILD_ID
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* [PATCH 25/25] perf record: Add --buildid-mmap option to enable mmap's build id
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (23 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 24/25] perf buildid-list: Add support for mmap2's buildid events Jiri Olsa
@ 2020-11-23 23:05 ` Jiri Olsa
  2020-11-25 12:54 ` [PATCHv3 00/25] perf: Add mmap2 build id support Namhyung Kim
  25 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-23 23:05 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: Ian Rogers, lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland,
	Namhyung Kim, Alexander Shishkin, Michael Petlan, Song Liu,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Adding --buildid-mmap option to enable build id in mmap2 events.
It will only work if there's kernel support for that and it disables
build id cache (implies --no-buildid).

It's also possible to enable it permanently via config option
in ~.perfconfig file:

  [record]
  build-id=mmap

Also added build_id bit in the verbose output for perf_event_attr:

  # perf record --buildid-mmap -vv
  ...
  perf_event_attr:
    type                             1
    size                             120
    ...
    build_id                         1

Adding also missing text_poke bit.

Acked-by: Ian Rogers <irogers@google.com>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/Documentation/perf-config.txt  |  3 ++-
 tools/perf/Documentation/perf-record.txt  |  3 +++
 tools/perf/builtin-record.c               | 20 ++++++++++++++++++++
 tools/perf/util/evsel.c                   | 10 ++++++----
 tools/perf/util/perf_api_probe.c          | 10 ++++++++++
 tools/perf/util/perf_api_probe.h          |  1 +
 tools/perf/util/perf_event_attr_fprintf.c |  2 ++
 tools/perf/util/record.h                  |  1 +
 8 files changed, 45 insertions(+), 5 deletions(-)

diff --git a/tools/perf/Documentation/perf-config.txt b/tools/perf/Documentation/perf-config.txt
index e3672c5d801b..8a1c6c16821a 100644
--- a/tools/perf/Documentation/perf-config.txt
+++ b/tools/perf/Documentation/perf-config.txt
@@ -559,11 +559,12 @@ kmem.*::
 
 record.*::
 	record.build-id::
-		This option can be 'cache', 'no-cache' or 'skip'.
+		This option can be 'cache', 'no-cache', 'skip' or 'mmap'.
 		'cache' is to post-process data and save/update the binaries into
 		the build-id cache (in ~/.debug). This is the default.
 		But if this option is 'no-cache', it will not update the build-id cache.
 		'skip' skips post-processing and does not update the cache.
+		'mmap' skips post-processing and reads build-ids from MMAP events.
 
 	record.call-graph::
 		This is identical to 'call-graph.record-mode', except it is
diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 768888b9326a..1bcf51e24979 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -482,6 +482,9 @@ Specify vmlinux path which has debuginfo.
 --buildid-all::
 Record build-id of all DSOs regardless whether it's actually hit or not.
 
+--buildid-mmap::
+Record build ids in mmap2 events, disables build id cache (implies --no-buildid).
+
 --aio[=n]::
 Use <n> control blocks in asynchronous (Posix AIO) trace writing mode (default: 1, max: 4).
 Asynchronous mode is supported only when linking Perf tool with libc library
diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index adf311d15d3d..47bae9d82d43 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -102,6 +102,7 @@ struct record {
 	bool			no_buildid_cache;
 	bool			no_buildid_cache_set;
 	bool			buildid_all;
+	bool			buildid_mmap;
 	bool			timestamp_filename;
 	bool			timestamp_boundary;
 	struct switch_output	switch_output;
@@ -2139,6 +2140,8 @@ static int perf_record_config(const char *var, const char *value, void *cb)
 			rec->no_buildid_cache = true;
 		else if (!strcmp(value, "skip"))
 			rec->no_buildid = true;
+		else if (!strcmp(value, "mmap"))
+			rec->buildid_mmap = true;
 		else
 			return -1;
 		return 0;
@@ -2554,6 +2557,8 @@ static struct option __record_options[] = {
 		   "file", "vmlinux pathname"),
 	OPT_BOOLEAN(0, "buildid-all", &record.buildid_all,
 		    "Record build-id of all DSOs regardless of hits"),
+	OPT_BOOLEAN(0, "buildid-mmap", &record.buildid_mmap,
+		    "Record build-id in map events"),
 	OPT_BOOLEAN(0, "timestamp-filename", &record.timestamp_filename,
 		    "append timestamp to output filename"),
 	OPT_BOOLEAN(0, "timestamp-boundary", &record.timestamp_boundary,
@@ -2657,6 +2662,21 @@ int cmd_record(int argc, const char **argv)
 
 	}
 
+	if (rec->buildid_mmap) {
+		if (!perf_can_record_build_id()) {
+			pr_err("Failed: no support to record build id in mmap events, update your kernel.\n");
+			err = -EINVAL;
+			goto out_opts;
+		}
+		pr_debug("Enabling build id in mmap2 events.\n");
+		/* Enable mmap build id synthesizing. */
+		symbol_conf.buildid_mmap2 = true;
+		/* Enable perf_event_attr::build_id bit. */
+		rec->opts.build_id = true;
+		/* Disable build id cache. */
+		rec->no_buildid = true;
+	}
+
 	if (rec->opts.kcore)
 		rec->data.is_dir = true;
 
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index 1cad6051d8b0..749d806ee1d1 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -1170,10 +1170,12 @@ void evsel__config(struct evsel *evsel, struct record_opts *opts,
 	if (opts->sample_weight)
 		evsel__set_sample_bit(evsel, WEIGHT);
 
-	attr->task  = track;
-	attr->mmap  = track;
-	attr->mmap2 = track && !perf_missing_features.mmap2;
-	attr->comm  = track;
+	attr->task     = track;
+	attr->mmap     = track;
+	attr->mmap2    = track && !perf_missing_features.mmap2;
+	attr->comm     = track;
+	attr->build_id = track && opts->build_id;
+
 	/*
 	 * ksymbol is tracked separately with text poke because it needs to be
 	 * system wide and enabled immediately.
diff --git a/tools/perf/util/perf_api_probe.c b/tools/perf/util/perf_api_probe.c
index 3840d02f0f7b..829af17a0867 100644
--- a/tools/perf/util/perf_api_probe.c
+++ b/tools/perf/util/perf_api_probe.c
@@ -98,6 +98,11 @@ static void perf_probe_text_poke(struct evsel *evsel)
 	evsel->core.attr.text_poke = 1;
 }
 
+static void perf_probe_build_id(struct evsel *evsel)
+{
+	evsel->core.attr.build_id = 1;
+}
+
 bool perf_can_sample_identifier(void)
 {
 	return perf_probe_api(perf_probe_sample_identifier);
@@ -172,3 +177,8 @@ bool perf_can_aux_sample(void)
 
 	return true;
 }
+
+bool perf_can_record_build_id(void)
+{
+	return perf_probe_api(perf_probe_build_id);
+}
diff --git a/tools/perf/util/perf_api_probe.h b/tools/perf/util/perf_api_probe.h
index d5506a983a94..f12ca55f509a 100644
--- a/tools/perf/util/perf_api_probe.h
+++ b/tools/perf/util/perf_api_probe.h
@@ -11,5 +11,6 @@ bool perf_can_record_cpu_wide(void);
 bool perf_can_record_switch_events(void);
 bool perf_can_record_text_poke_events(void);
 bool perf_can_sample_identifier(void);
+bool perf_can_record_build_id(void);
 
 #endif // __PERF_API_PROBE_H
diff --git a/tools/perf/util/perf_event_attr_fprintf.c b/tools/perf/util/perf_event_attr_fprintf.c
index e67a227c0ce7..656a7fddfc26 100644
--- a/tools/perf/util/perf_event_attr_fprintf.c
+++ b/tools/perf/util/perf_event_attr_fprintf.c
@@ -134,6 +134,8 @@ int perf_event_attr__fprintf(FILE *fp, struct perf_event_attr *attr,
 	PRINT_ATTRf(bpf_event, p_unsigned);
 	PRINT_ATTRf(aux_output, p_unsigned);
 	PRINT_ATTRf(cgroup, p_unsigned);
+	PRINT_ATTRf(text_poke, p_unsigned);
+	PRINT_ATTRf(build_id, p_unsigned);
 
 	PRINT_ATTRn("{ wakeup_events, wakeup_watermark }", wakeup_events, p_unsigned);
 	PRINT_ATTRf(bp_type, p_unsigned);
diff --git a/tools/perf/util/record.h b/tools/perf/util/record.h
index 266760ac9143..609e706f4282 100644
--- a/tools/perf/util/record.h
+++ b/tools/perf/util/record.h
@@ -49,6 +49,7 @@ struct record_opts {
 	bool	      no_bpf_event;
 	bool	      kcore;
 	bool	      text_poke;
+	bool	      build_id;
 	unsigned int  freq;
 	unsigned int  mmap_pages;
 	unsigned int  auxtrace_mmap_pages;
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

* Re: [PATCHv3 00/25] perf: Add mmap2 build id support
  2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
                   ` (24 preceding siblings ...)
  2020-11-23 23:05 ` [PATCH 25/25] perf record: Add --buildid-mmap option to enable mmap's build id Jiri Olsa
@ 2020-11-25 12:54 ` Namhyung Kim
  25 siblings, 0 replies; 37+ messages in thread
From: Namhyung Kim @ 2020-11-25 12:54 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, Frank Ch . Eigler, Mark Wielaard, lkml,
	Peter Zijlstra, Ingo Molnar, Mark Rutland, Alexander Shishkin,
	Michael Petlan, Song Liu, Ian Rogers, Stephane Eranian,
	Alexey Budankov, Andi Kleen, Adrian Hunter

Hi Jiri,

On Tue, Nov 24, 2020 at 8:05 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> hi,
> adding the support to have buildid stored in mmap2 event,
> so we can bypass the final perf record hunt on build ids.
>
> This patchset allows perf to record build ID in mmap2 event,
> and adds perf tooling to store/download binaries to .debug
> cache based on these build IDs.
>
> Note that the build id retrieval code is stolen from bpf
> code, where it's been used (together with file offsets)
> to replace IPs in user space stack traces. It's now added
> under lib directory.
>
> v3 changes:
>   - added acks
>   - removed forgotten debug code [Arnaldo]
>   - fixed readlink termination [Ian]
>   - fixed doc for --debuginfod=URLs [Ian]
>   - adopted kernel's memchr_inv function and used
>     it in build_id__is_defined function [Arnaldo]

[SNIP]
> ---
> Jiri Olsa (25):
>       bpf: Move stack_map_get_build_id into lib
>       bpf: Add size arg to build_id_parse function
>       perf: Add build id data in mmap2 event
>       tools headers uapi: Sync tools/include/uapi/linux/perf_event.h
>       tools lib: Adopt memchr_inv() from kernel
>       perf tools: Do not swap mmap2 fields in case it contains build id
>       perf tools: Add build_id__is_defined function
>       perf tools: Add filename__decompress function
>       perf tools: Add support to read build id from compressed elf
>       perf tools: Add check for existing link in buildid dir
>       perf tools: Use struct extra_kernel_map in machine__process_kernel_mmap_event
>       perf tools: Try to load vmlinux from buildid database
>       perf tools: Store build id from mmap2 events
>       perf tools: Allow mmap2 event to synthesize kernel image
>       perf tools: Allow mmap2 event to synthesize modules
>       perf tools: Synthesize build id for kernel/modules/tasks
>       perf tools: Add support to display build id for mmap2 events
>       perf tools: Use machine__for_each_dso in perf_session__cache_build_ids
>       perf tools: Add __perf_session__cache_build_ids function
>       perf tools: Add is_perf_data function
>       perf tools: Add build_id_cache__add function
>       perf buildid-cache: Add support to add build ids from perf data
>       perf buildid-cache: Add --debuginfod option
>       perf buildid-list: Add support for mmap2's buildid events
>       perf record: Add --buildid-mmap option to enable mmap's build id

Mostly looks good!

I only have some comments on the buildid-cache part.

Thanks,
Namhyung

>
>  include/linux/buildid.h                           |  12 +++++
>  include/uapi/linux/perf_event.h                   |  42 +++++++++++++++---
>  kernel/bpf/stackmap.c                             | 143 ++---------------------------------------------------------
>  kernel/events/core.c                              |  32 ++++++++++++--
>  lib/Makefile                                      |   3 +-
>  lib/buildid.c                                     | 149 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tools/include/linux/string.h                      |   1 +
>  tools/include/uapi/linux/perf_event.h             |  42 +++++++++++++++---
>  tools/lib/perf/include/perf/event.h               |  18 ++++++--
>  tools/lib/string.c                                |  58 ++++++++++++++++++++++++
>  tools/perf/Documentation/perf-buildid-cache.txt   |  18 +++++++-
>  tools/perf/Documentation/perf-config.txt          |  10 ++++-
>  tools/perf/Documentation/perf-record.txt          |   3 ++
>  tools/perf/builtin-buildid-cache.c                | 243 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
>  tools/perf/builtin-buildid-list.c                 |   3 ++
>  tools/perf/builtin-record.c                       |  20 +++++++++
>  tools/perf/tests/shell/trace+probe_vfs_getname.sh |   2 +-
>  tools/perf/util/build-id.c                        | 127 ++++++++++++++++++++++++++++++++++-------------------
>  tools/perf/util/build-id.h                        |   8 ++++
>  tools/perf/util/data.c                            |  19 ++++++++
>  tools/perf/util/data.h                            |   1 +
>  tools/perf/util/dso.c                             |  31 ++++++++-----
>  tools/perf/util/dso.h                             |   2 +
>  tools/perf/util/event.c                           |  41 ++++++++++++-----
>  tools/perf/util/evsel.c                           |  10 +++--
>  tools/perf/util/machine.c                         |  80 ++++++++++++++++++++-------------
>  tools/perf/util/map.c                             |   8 +++-
>  tools/perf/util/map.h                             |   3 +-
>  tools/perf/util/perf_api_probe.c                  |  10 +++++
>  tools/perf/util/perf_api_probe.h                  |   1 +
>  tools/perf/util/perf_event_attr_fprintf.c         |   2 +
>  tools/perf/util/probe-event.c                     |   6 +--
>  tools/perf/util/record.h                          |   1 +
>  tools/perf/util/session.c                         |  11 +++--
>  tools/perf/util/symbol-elf.c                      |  37 +++++++++++++++-
>  tools/perf/util/symbol.c                          |  16 +++++++
>  tools/perf/util/symbol_conf.h                     |   3 +-
>  tools/perf/util/synthetic-events.c                | 121 +++++++++++++++++++++++++++++++++++++-------------
>  38 files changed, 1026 insertions(+), 311 deletions(-)
>  create mode 100644 include/linux/buildid.h
>  create mode 100644 lib/buildid.c
>

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 13/25] perf tools: Store build id from mmap2 events
  2020-11-23 23:05 ` [PATCH 13/25] perf tools: Store build id from mmap2 events Jiri Olsa
@ 2020-11-25 12:56   ` Namhyung Kim
  2020-11-25 14:15     ` Jiri Olsa
  0 siblings, 1 reply; 37+ messages in thread
From: Namhyung Kim @ 2020-11-25 12:56 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, lkml, Peter Zijlstra, Ingo Molnar,
	Mark Rutland, Alexander Shishkin, Michael Petlan, Song Liu,
	Ian Rogers, Stephane Eranian, Alexey Budankov, Andi Kleen,
	Adrian Hunter

On Tue, Nov 24, 2020 at 8:06 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> When processing mmap2 event, check on the build id
> misc bit: PERF_RECORD_MISC_MMAP_BUILD_ID and if it
> is set, store the build id in mmap's dso object.
>
> Also adding the build id data arts to struct

s/arts/args/ ?

Thanks,
Namhyung


> perf_record_mmap2 event definition.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data
  2020-11-23 23:05 ` [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data Jiri Olsa
@ 2020-11-25 13:00   ` Namhyung Kim
  2020-11-25 13:04     ` Namhyung Kim
  2020-11-25 16:36     ` Jiri Olsa
  2020-11-25 13:29   ` Namhyung Kim
  1 sibling, 2 replies; 37+ messages in thread
From: Namhyung Kim @ 2020-11-25 13:00 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, lkml, Peter Zijlstra, Ingo Molnar,
	Mark Rutland, Alexander Shishkin, Michael Petlan, Song Liu,
	Ian Rogers, Stephane Eranian, Alexey Budankov, Andi Kleen,
	Adrian Hunter

On Tue, Nov 24, 2020 at 8:06 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding support to specify perf data file as -a option file
> argument,
>
> If the file is detected to be perf data file, it is processed
> and all dso objects with sample hit are stored to the build
> id cache.
>
>   $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data
>   OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
>   OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
>
> By default we store only dso with hits, but it's possible to
> specify 'all' to store all dso objects, like:
>     -a perf.data,all

I think we can add -A/--add-all like we have -p and -P.

>
>   $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data,all
>   OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
>   OK   6ce92dc7c31f12fe5b7775a2bb8b14a3546ce2cd /lib/modules/5.10.0-rc2speed+/kernel/drivers/firmware/qemu_fw_cfg.ko
>   OK   bf3f6d32dccc159f841fc3658c241d0e74c61fbb /lib/modules/5.10.0-rc2speed+/kernel/drivers/block/virtio_blk.ko
>   OK   e896b4329cf9f190f1a0fae933f425ff8f71b052 /lib/modules/5.10.0-rc2speed+/kernel/drivers/char/virtio_console.ko
>   OK   5bedc933cb59e053ecb472f327bd73c548364479 /lib/modules/5.10.0-rc2speed+/kernel/drivers/input/serio/serio_raw.ko
>   OK   cecc506368a8b7a473a5f900d26f0d3d914a9c23 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32c-intel.ko
>   OK   91076fb3646d061a0a42cf7bddb339a665ee4f80 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/ghash-clmulni-intel.ko
>   OK   4e2a304d788bb8e2e950bc82a5944e042afa0bf2 /lib/modules/5.10.0-rc2speed+/kernel/drivers/media/cec/core/cec.ko
>   OK   31ab0da5ad81e6803280177f507a95f3053d585e /lib/modules/5.10.0-rc2speed+/kernel/lib/libcrc32c.ko
>   OK   f6154bca47c149f48c942fcc3d653041dd285c65 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/ttm/ttm.ko
>   OK   723f5852de81590d54b23b38c160d3618b41951b /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crct10dif-pclmul.ko
>   OK   06b1eab7f141cbc3e5a5db47909c8ab5cb242e40 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_ttm_helper.ko
>   OK   38292b862cf3ff87489508fdb4895efa45780813 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/qxl/qxl.ko
>   OK   cdf51e58609bf2ce4837a7b195e0ccae0a930907 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32-pclmul.ko
>   OK   5ca8958388f6688452ecc2cb83d6031394c659ad /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm.ko
>   OK   236bc4e4f38bf3559007566cb32b3dcc1bc28d2d /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_kms_helper.ko
>   OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
>   OK   66db2be3efaa43bb5a5c481986e9554e1885cc69 /usr/lib/systemd/systemd
>   OK   7db607d9f2de89860d9639712da64c8bacd31e4b /usr/lib64/libm-2.30.so
>   OK   55b5f9652e1d17c1dd58f62628d5063428e5db91 /usr/lib64/libudev.so.1.6.15
>   OK   63b97070bf097130713bb6c89cf7100b5f3c9b17 /usr/lib64/libunistring.so.2.1.0
>   ...
>
> Once perf data is specified, no other file can be specified in
> the option, otherwise it causes syntax error.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  .../perf/Documentation/perf-buildid-cache.txt |  12 +-
>  tools/perf/builtin-buildid-cache.c            | 215 +++++++++++++++++-
>  tools/perf/util/probe-event.c                 |   6 +-
>  3 files changed, 227 insertions(+), 6 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-buildid-cache.txt b/tools/perf/Documentation/perf-buildid-cache.txt
> index f6de0952ff3c..b77da5138bca 100644
> --- a/tools/perf/Documentation/perf-buildid-cache.txt
> +++ b/tools/perf/Documentation/perf-buildid-cache.txt
> @@ -23,7 +23,17 @@ OPTIONS
>  -------
>  -a::
>  --add=::
> -        Add specified file to the cache.
> +        Add specified file or perf.data binaries to the cache.
> +
> +        If the file is detected to be perf data file, it is processed
> +        and all dso objects with sample hit are stored to the cache.
> +
> +        It's possible to specify 'all' to store all dso objects, like:
> +            -a perf.data,all
> +
> +        Once perf data is specified, no other file can be specified in
> +        the option, otherwise it causes syntax error.
> +
>  -f::
>  --force::
>         Don't complain, do it.
> diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
> index a25411926e48..0bfb54ee1e5e 100644
> --- a/tools/perf/builtin-buildid-cache.c
> +++ b/tools/perf/builtin-buildid-cache.c
> @@ -29,6 +29,11 @@
>  #include "util/probe-file.h"
>  #include <linux/string.h>
>  #include <linux/err.h>
> +#include <linux/zalloc.h>
> +#include <sys/stat.h>
> +#ifdef HAVE_DEBUGINFOD_SUPPORT
> +#include <elfutils/debuginfod.h>
> +#endif
>
>  static int build_id_cache__kcore_buildid(const char *proc_dir, char *sbuildid)
>  {
> @@ -348,6 +353,205 @@ static int build_id_cache__show_all(void)
>         return 0;
>  }
>
> +#ifdef HAVE_DEBUGINFOD_SUPPORT

I think this part belongs to the next patch...?

Thanks,
Namhyung


> +static int call_debuginfod(const char *sbuild_id, char **path, bool debuginfo)
> +{
> +       debuginfod_client *c;
> +       int fd;
> +
> +       c = debuginfod_begin();
> +       if (c == NULL)
> +               return -1;
> +
> +       pr_debug("trying debuginfod for executable <%s> ... ", sbuild_id);
> +
> +       if (debuginfo) {
> +               fd = debuginfod_find_debuginfo(c, (const unsigned char *) sbuild_id,
> +                                              0, path);
> +       } else {
> +               fd = debuginfod_find_executable(c, (const unsigned char *) sbuild_id,
> +                                               0, path);
> +       }
> +       if (fd >= 0)
> +               close(fd); /* retaining reference by realname */
> +
> +       debuginfod_end(c);
> +       pr_debug("%s%s\n", *path ? "OK " : "FAILED", *path ? *path : "");
> +       return *path ? 0 : -1;
> +}
> +#else
> +static int call_debuginfod(const char *sbuild_id __maybe_unused,
> +                          char **path __maybe_unused,
> +                          bool debuginfo __maybe_unused)
> +{
> +       return -1;
> +}
> +#endif
> +
> +struct dso_store_data {
> +       bool     hits;
> +       bool     force_download;
> +};
> +
> +static int dso_store(struct dso *dso, struct machine *machine, void *priv)
> +{
> +       struct dso_store_data *data = priv;
> +       char sbuild_id[SBUILD_ID_SIZE];
> +       struct build_id bid;
> +       char *path = NULL, *link = NULL;
> +       bool is_kallsyms;
> +       int err = -1;
> +
> +       /*
> +        * There's no build id in dso, nothing to do..
> +        */
> +       if (!dso->has_build_id || !build_id__is_defined(&dso->bid))
> +               return 0;
> +
> +       if (data->hits && !dso->hit)
> +               return 0;
> +
> +       /*
> +        * The storing process is:
> +        *   - get build id of the dso
> +        *   - check if it is already in cache
> +        *   - check if it matches provided build id from mmap2 event
> +        *   - if not, try debuginfod to download the binary
> +        *   - store binary to build id database
> +        */
> +       is_kallsyms = !strcmp(machine->mmap_name, dso->short_name);
> +       build_id__sprintf(&dso->bid, sbuild_id);
> +
> +       link = build_id_cache__linkname(sbuild_id, NULL, 0);
> +       if (!link)
> +               return -ENOMEM;
> +
> +       if (!data->force_download && !access(link, X_OK)) {
> +               pr_debug("already in cache - %s <%s>\n", dso->long_name, sbuild_id);
> +               err = 0;
> +               goto out;
> +       }
> +
> +       path = strdup(dso->long_name);
> +       if (!path)
> +               goto out;
> +
> +       if (is_kallsyms) {
> +               /*
> +                * Find out if we are on the same kernel as perf.data
> +                * and store kallsyms in that case.
> +                */
> +               err = sysfs__read_build_id("/sys/kernel/notes", &bid);
> +               if (err < 0)
> +                       goto out;
> +       } else {
> +               struct nscookie nsc;
> +               struct stat st;
> +
> +               nsinfo__mountns_enter(dso->nsinfo, &nsc);
> +
> +               /*
> +                * Does the file exists in the first place, if it does,
> +                * resolve path and read the build id.
> +                */
> +               if (stat(dso->long_name, &st)) {
> +                       nsinfo__mountns_exit(&nsc);
> +                       zfree(&path);
> +                       goto try_download;
> +               }
> +
> +               err = filename__read_build_id(dso->long_name, &bid);
> +               nsinfo__mountns_exit(&nsc);
> +
> +               if (err <= 0)
> +                       goto out;
> +       }
> +
> +       /*
> +        * If we match, then what we want in mmap2 event
> +        * is what we got in the binary,
> +        */
> +       if (bid.size != dso->bid.size || memcmp(&bid, &dso->bid, bid.size)) {
> +               char sbid[SBUILD_ID_SIZE];
> +
> +               build_id__sprintf(&bid, sbid);
> +               pr_debug("mmap build id <%s> does not match for %s <%s>\n",
> +                        sbuild_id, path, sbid);
> +               zfree(&path);
> +       }
> +
> +try_download:
> +       /*
> +        * We did not match build id or did not find the
> +        * binary - try debuginfod as last resort.
> +        */
> +       if (!path) {
> +               bool debuginfo;
> +               char *tmp = NULL;
> +
> +               /*
> +                * The debuginfo retrieval for standard binaries
> +                * is handled within build_id_cache__add function.
> +                *
> +                * For kernel and kernel modules we have to ask
> +                * for debuginfo directly, because debuginfod
> +                * does not treat them as binaries.
> +                */
> +               debuginfo = is_kallsyms ||
> +                           is_kernel_module(dso->long_name, PERF_RECORD_MISC_CPUMODE_UNKNOWN);
> +
> +               if (call_debuginfod(sbuild_id, &tmp, debuginfo)) {
> +                       err = -1;
> +                       goto out;
> +               }
> +
> +               path = tmp;
> +
> +               /*
> +                * The kernel dso is now elf binary, so disable is_kallsyms
> +                * so build_id_cache__add can prepare proper file names.
> +                */
> +               is_kallsyms = false;
> +       }
> +
> +       pr_debug("linking %s %s <%s>\n", dso->short_name, path, sbuild_id);
> +
> +       err = build_id_cache__add(sbuild_id, path, path,
> +                                 dso->nsinfo, is_kallsyms, false);
> +out:
> +       free(path);
> +       fprintf(stderr, "%s %s %s\n", err ? "FAIL" : "OK  ", sbuild_id, dso->long_name);
> +       return 0;
> +}
> +
> +static int
> +build_id_cache__add_perf_data(const char *path, bool all)
> +{
> +       struct perf_session *session;
> +       struct dso_store_data priv = {
> +               .hits = !all,
> +               .force_download = false,
> +       };
> +       struct perf_data data = {
> +               .path  = path,
> +               .mode  = PERF_DATA_MODE_READ,
> +       };
> +       int err;
> +
> +       session = perf_session__new(&data, false, &build_id__mark_dso_hit_ops);
> +       if (IS_ERR(session))
> +               return PTR_ERR(session);
> +
> +       err = perf_session__process_events(session);
> +       if (err)
> +               goto out;
> +
> +       err = __perf_session__cache_build_ids(session, dso_store, &priv);
> +out:
> +       perf_session__delete(session);
> +       return err;
> +}
> +
>  int cmd_buildid_cache(int argc, const char **argv)
>  {
>         struct strlist *list;
> @@ -440,7 +644,15 @@ int cmd_buildid_cache(int argc, const char **argv)
>                 list = strlist__new(add_name_list_str, NULL);
>                 if (list) {
>                         strlist__for_each_entry(pos, list)
> -                               if (build_id_cache__add_file(pos->s, nsi)) {
> +                               if (is_perf_data(pos->s)) {
> +                                       struct str_node *all_pos = strlist__next(pos);
> +                                       bool all = !strcmp("all", all_pos ? all_pos->s : "");
> +
> +                                       if (build_id_cache__add_perf_data(pos->s, all))
> +                                               pr_warning("Couldn't add build ids from %s\n", pos->s);
> +                                       if (all)
> +                                               pos = all_pos;
> +                               } else if (build_id_cache__add_file(pos->s, nsi)) {
>                                         if (errno == EEXIST) {
>                                                 pr_debug("%s already in the cache\n",
>                                                          pos->s);
> @@ -449,7 +661,6 @@ int cmd_buildid_cache(int argc, const char **argv)
>                                         pr_warning("Couldn't add %s: %s\n",
>                                                    pos->s, str_error_r(errno, sbuf, sizeof(sbuf)));
>                                 }
> -
>                         strlist__delete(list);
>                 }
>         }
> diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> index 8eae2afff71a..e821bb977c9b 100644
> --- a/tools/perf/util/probe-event.c
> +++ b/tools/perf/util/probe-event.c
> @@ -1616,9 +1616,9 @@ static int parse_perf_probe_point(char *arg, struct perf_probe_event *pev)
>                 return -EINVAL;
>         }
>
> -       pr_debug("symbol:%s file:%s line:%d offset:%lu return:%d lazy:%s\n",
> -                pp->function, pp->file, pp->line, pp->offset, pp->retprobe,
> -                pp->lazy_line);
> +       pr_debug2("symbol:%s file:%s line:%d offset:%lu return:%d lazy:%s\n",
> +                 pp->function, pp->file, pp->line, pp->offset, pp->retprobe,
> +                 pp->lazy_line);
>         return 0;
>  }
>
> --
> 2.26.2
>

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data
  2020-11-25 13:00   ` Namhyung Kim
@ 2020-11-25 13:04     ` Namhyung Kim
  2020-11-25 16:36     ` Jiri Olsa
  1 sibling, 0 replies; 37+ messages in thread
From: Namhyung Kim @ 2020-11-25 13:04 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, lkml, Peter Zijlstra, Ingo Molnar,
	Mark Rutland, Alexander Shishkin, Michael Petlan, Song Liu,
	Ian Rogers, Stephane Eranian, Alexey Budankov, Andi Kleen,
	Adrian Hunter

On Wed, Nov 25, 2020 at 10:00 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Tue, Nov 24, 2020 at 8:06 AM Jiri Olsa <jolsa@kernel.org> wrote:
> > diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
> > index a25411926e48..0bfb54ee1e5e 100644
> > --- a/tools/perf/builtin-buildid-cache.c
> > +++ b/tools/perf/builtin-buildid-cache.c
> > @@ -29,6 +29,11 @@
> >  #include "util/probe-file.h"
> >  #include <linux/string.h>
> >  #include <linux/err.h>
> > +#include <linux/zalloc.h>
> > +#include <sys/stat.h>
> > +#ifdef HAVE_DEBUGINFOD_SUPPORT
> > +#include <elfutils/debuginfod.h>
> > +#endif
> >
> >  static int build_id_cache__kcore_buildid(const char *proc_dir, char *sbuildid)
> >  {
> > @@ -348,6 +353,205 @@ static int build_id_cache__show_all(void)
> >         return 0;
> >  }
> >
> > +#ifdef HAVE_DEBUGINFOD_SUPPORT
>
> I think this part belongs to the next patch...?
>
> Thanks,
> Namhyung

Oh, I've missed we already have the debuginfod support.

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data
  2020-11-23 23:05 ` [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data Jiri Olsa
  2020-11-25 13:00   ` Namhyung Kim
@ 2020-11-25 13:29   ` Namhyung Kim
  2020-11-25 14:20     ` Jiri Olsa
  1 sibling, 1 reply; 37+ messages in thread
From: Namhyung Kim @ 2020-11-25 13:29 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Arnaldo Carvalho de Melo, lkml, Peter Zijlstra, Ingo Molnar,
	Mark Rutland, Alexander Shishkin, Michael Petlan, Song Liu,
	Ian Rogers, Stephane Eranian, Alexey Budankov, Andi Kleen,
	Adrian Hunter

On Tue, Nov 24, 2020 at 8:06 AM Jiri Olsa <jolsa@kernel.org> wrote:
>
> Adding support to specify perf data file as -a option file
> argument,
>
> If the file is detected to be perf data file, it is processed
> and all dso objects with sample hit are stored to the build
> id cache.
>
>   $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data
>   OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
>   OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
>
> By default we store only dso with hits, but it's possible to
> specify 'all' to store all dso objects, like:
>     -a perf.data,all
>
>   $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data,all
>   OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
>   OK   6ce92dc7c31f12fe5b7775a2bb8b14a3546ce2cd /lib/modules/5.10.0-rc2speed+/kernel/drivers/firmware/qemu_fw_cfg.ko
>   OK   bf3f6d32dccc159f841fc3658c241d0e74c61fbb /lib/modules/5.10.0-rc2speed+/kernel/drivers/block/virtio_blk.ko
>   OK   e896b4329cf9f190f1a0fae933f425ff8f71b052 /lib/modules/5.10.0-rc2speed+/kernel/drivers/char/virtio_console.ko
>   OK   5bedc933cb59e053ecb472f327bd73c548364479 /lib/modules/5.10.0-rc2speed+/kernel/drivers/input/serio/serio_raw.ko
>   OK   cecc506368a8b7a473a5f900d26f0d3d914a9c23 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32c-intel.ko
>   OK   91076fb3646d061a0a42cf7bddb339a665ee4f80 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/ghash-clmulni-intel.ko
>   OK   4e2a304d788bb8e2e950bc82a5944e042afa0bf2 /lib/modules/5.10.0-rc2speed+/kernel/drivers/media/cec/core/cec.ko
>   OK   31ab0da5ad81e6803280177f507a95f3053d585e /lib/modules/5.10.0-rc2speed+/kernel/lib/libcrc32c.ko
>   OK   f6154bca47c149f48c942fcc3d653041dd285c65 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/ttm/ttm.ko
>   OK   723f5852de81590d54b23b38c160d3618b41951b /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crct10dif-pclmul.ko
>   OK   06b1eab7f141cbc3e5a5db47909c8ab5cb242e40 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_ttm_helper.ko
>   OK   38292b862cf3ff87489508fdb4895efa45780813 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/qxl/qxl.ko
>   OK   cdf51e58609bf2ce4837a7b195e0ccae0a930907 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32-pclmul.ko
>   OK   5ca8958388f6688452ecc2cb83d6031394c659ad /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm.ko
>   OK   236bc4e4f38bf3559007566cb32b3dcc1bc28d2d /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_kms_helper.ko
>   OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
>   OK   66db2be3efaa43bb5a5c481986e9554e1885cc69 /usr/lib/systemd/systemd
>   OK   7db607d9f2de89860d9639712da64c8bacd31e4b /usr/lib64/libm-2.30.so
>   OK   55b5f9652e1d17c1dd58f62628d5063428e5db91 /usr/lib64/libudev.so.1.6.15
>   OK   63b97070bf097130713bb6c89cf7100b5f3c9b17 /usr/lib64/libunistring.so.2.1.0
>   ...
>
> Once perf data is specified, no other file can be specified in
> the option, otherwise it causes syntax error.
>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  .../perf/Documentation/perf-buildid-cache.txt |  12 +-
>  tools/perf/builtin-buildid-cache.c            | 215 +++++++++++++++++-
>  tools/perf/util/probe-event.c                 |   6 +-
>  3 files changed, 227 insertions(+), 6 deletions(-)
>
> diff --git a/tools/perf/Documentation/perf-buildid-cache.txt b/tools/perf/Documentation/perf-buildid-cache.txt
> index f6de0952ff3c..b77da5138bca 100644
> --- a/tools/perf/Documentation/perf-buildid-cache.txt
> +++ b/tools/perf/Documentation/perf-buildid-cache.txt
> @@ -23,7 +23,17 @@ OPTIONS
>  -------
>  -a::
>  --add=::
> -        Add specified file to the cache.
> +        Add specified file or perf.data binaries to the cache.
> +
> +        If the file is detected to be perf data file, it is processed
> +        and all dso objects with sample hit are stored to the cache.
> +
> +        It's possible to specify 'all' to store all dso objects, like:
> +            -a perf.data,all
> +
> +        Once perf data is specified, no other file can be specified in
> +        the option, otherwise it causes syntax error.
> +
>  -f::
>  --force::
>         Don't complain, do it.
> diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
> index a25411926e48..0bfb54ee1e5e 100644
> --- a/tools/perf/builtin-buildid-cache.c
> +++ b/tools/perf/builtin-buildid-cache.c
> @@ -29,6 +29,11 @@
>  #include "util/probe-file.h"
>  #include <linux/string.h>
>  #include <linux/err.h>
> +#include <linux/zalloc.h>
> +#include <sys/stat.h>
> +#ifdef HAVE_DEBUGINFOD_SUPPORT
> +#include <elfutils/debuginfod.h>
> +#endif
>
>  static int build_id_cache__kcore_buildid(const char *proc_dir, char *sbuildid)
>  {
> @@ -348,6 +353,205 @@ static int build_id_cache__show_all(void)
>         return 0;
>  }
>
> +#ifdef HAVE_DEBUGINFOD_SUPPORT
> +static int call_debuginfod(const char *sbuild_id, char **path, bool debuginfo)
> +{
> +       debuginfod_client *c;
> +       int fd;
> +
> +       c = debuginfod_begin();
> +       if (c == NULL)
> +               return -1;
> +
> +       pr_debug("trying debuginfod for executable <%s> ... ", sbuild_id);
> +
> +       if (debuginfo) {
> +               fd = debuginfod_find_debuginfo(c, (const unsigned char *) sbuild_id,
> +                                              0, path);
> +       } else {
> +               fd = debuginfod_find_executable(c, (const unsigned char *) sbuild_id,
> +                                               0, path);
> +       }
> +       if (fd >= 0)
> +               close(fd); /* retaining reference by realname */
> +
> +       debuginfod_end(c);
> +       pr_debug("%s%s\n", *path ? "OK " : "FAILED", *path ? *path : "");
> +       return *path ? 0 : -1;
> +}
> +#else
> +static int call_debuginfod(const char *sbuild_id __maybe_unused,
> +                          char **path __maybe_unused,
> +                          bool debuginfo __maybe_unused)
> +{
> +       return -1;
> +}
> +#endif
> +
> +struct dso_store_data {
> +       bool     hits;
> +       bool     force_download;

Where is it set?

Thanks,
Namhyung


> +};
> +
> +static int dso_store(struct dso *dso, struct machine *machine, void *priv)
> +{
> +       struct dso_store_data *data = priv;
> +       char sbuild_id[SBUILD_ID_SIZE];
> +       struct build_id bid;
> +       char *path = NULL, *link = NULL;
> +       bool is_kallsyms;
> +       int err = -1;
> +
> +       /*
> +        * There's no build id in dso, nothing to do..
> +        */
> +       if (!dso->has_build_id || !build_id__is_defined(&dso->bid))
> +               return 0;
> +
> +       if (data->hits && !dso->hit)
> +               return 0;
> +
> +       /*
> +        * The storing process is:
> +        *   - get build id of the dso
> +        *   - check if it is already in cache
> +        *   - check if it matches provided build id from mmap2 event
> +        *   - if not, try debuginfod to download the binary
> +        *   - store binary to build id database
> +        */
> +       is_kallsyms = !strcmp(machine->mmap_name, dso->short_name);
> +       build_id__sprintf(&dso->bid, sbuild_id);
> +
> +       link = build_id_cache__linkname(sbuild_id, NULL, 0);
> +       if (!link)
> +               return -ENOMEM;
> +
> +       if (!data->force_download && !access(link, X_OK)) {
> +               pr_debug("already in cache - %s <%s>\n", dso->long_name, sbuild_id);
> +               err = 0;
> +               goto out;
> +       }
> +
> +       path = strdup(dso->long_name);
> +       if (!path)
> +               goto out;
> +
> +       if (is_kallsyms) {
> +               /*
> +                * Find out if we are on the same kernel as perf.data
> +                * and store kallsyms in that case.
> +                */
> +               err = sysfs__read_build_id("/sys/kernel/notes", &bid);
> +               if (err < 0)
> +                       goto out;
> +       } else {
> +               struct nscookie nsc;
> +               struct stat st;
> +
> +               nsinfo__mountns_enter(dso->nsinfo, &nsc);
> +
> +               /*
> +                * Does the file exists in the first place, if it does,
> +                * resolve path and read the build id.
> +                */
> +               if (stat(dso->long_name, &st)) {
> +                       nsinfo__mountns_exit(&nsc);
> +                       zfree(&path);
> +                       goto try_download;
> +               }
> +
> +               err = filename__read_build_id(dso->long_name, &bid);
> +               nsinfo__mountns_exit(&nsc);
> +
> +               if (err <= 0)
> +                       goto out;
> +       }
> +
> +       /*
> +        * If we match, then what we want in mmap2 event
> +        * is what we got in the binary,
> +        */
> +       if (bid.size != dso->bid.size || memcmp(&bid, &dso->bid, bid.size)) {
> +               char sbid[SBUILD_ID_SIZE];
> +
> +               build_id__sprintf(&bid, sbid);
> +               pr_debug("mmap build id <%s> does not match for %s <%s>\n",
> +                        sbuild_id, path, sbid);
> +               zfree(&path);
> +       }
> +
> +try_download:
> +       /*
> +        * We did not match build id or did not find the
> +        * binary - try debuginfod as last resort.
> +        */
> +       if (!path) {
> +               bool debuginfo;
> +               char *tmp = NULL;
> +
> +               /*
> +                * The debuginfo retrieval for standard binaries
> +                * is handled within build_id_cache__add function.
> +                *
> +                * For kernel and kernel modules we have to ask
> +                * for debuginfo directly, because debuginfod
> +                * does not treat them as binaries.
> +                */
> +               debuginfo = is_kallsyms ||
> +                           is_kernel_module(dso->long_name, PERF_RECORD_MISC_CPUMODE_UNKNOWN);
> +
> +               if (call_debuginfod(sbuild_id, &tmp, debuginfo)) {
> +                       err = -1;
> +                       goto out;
> +               }
> +
> +               path = tmp;
> +
> +               /*
> +                * The kernel dso is now elf binary, so disable is_kallsyms
> +                * so build_id_cache__add can prepare proper file names.
> +                */
> +               is_kallsyms = false;
> +       }
> +
> +       pr_debug("linking %s %s <%s>\n", dso->short_name, path, sbuild_id);
> +
> +       err = build_id_cache__add(sbuild_id, path, path,
> +                                 dso->nsinfo, is_kallsyms, false);
> +out:
> +       free(path);
> +       fprintf(stderr, "%s %s %s\n", err ? "FAIL" : "OK  ", sbuild_id, dso->long_name);
> +       return 0;
> +}

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 13/25] perf tools: Store build id from mmap2 events
  2020-11-25 12:56   ` Namhyung Kim
@ 2020-11-25 14:15     ` Jiri Olsa
  0 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-25 14:15 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Jiri Olsa, Arnaldo Carvalho de Melo, lkml, Peter Zijlstra,
	Ingo Molnar, Mark Rutland, Alexander Shishkin, Michael Petlan,
	Song Liu, Ian Rogers, Stephane Eranian, Alexey Budankov,
	Andi Kleen, Adrian Hunter

On Wed, Nov 25, 2020 at 09:56:31PM +0900, Namhyung Kim wrote:
> On Tue, Nov 24, 2020 at 8:06 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > When processing mmap2 event, check on the build id
> > misc bit: PERF_RECORD_MISC_MMAP_BUILD_ID and if it
> > is set, store the build id in mmap's dso object.
> >
> > Also adding the build id data arts to struct
> 
> s/arts/args/ ?

right, perhaps also s/arts// 

thanks,
jirka

> 
> Thanks,
> Namhyung
> 
> 
> > perf_record_mmap2 event definition.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> 


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data
  2020-11-25 13:29   ` Namhyung Kim
@ 2020-11-25 14:20     ` Jiri Olsa
  0 siblings, 0 replies; 37+ messages in thread
From: Jiri Olsa @ 2020-11-25 14:20 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Jiri Olsa, Arnaldo Carvalho de Melo, lkml, Peter Zijlstra,
	Ingo Molnar, Mark Rutland, Alexander Shishkin, Michael Petlan,
	Song Liu, Ian Rogers, Stephane Eranian, Alexey Budankov,
	Andi Kleen, Adrian Hunter

On Wed, Nov 25, 2020 at 10:29:11PM +0900, Namhyung Kim wrote:

SNIP

> > +#ifdef HAVE_DEBUGINFOD_SUPPORT
> > +static int call_debuginfod(const char *sbuild_id, char **path, bool debuginfo)
> > +{
> > +       debuginfod_client *c;
> > +       int fd;
> > +
> > +       c = debuginfod_begin();
> > +       if (c == NULL)
> > +               return -1;
> > +
> > +       pr_debug("trying debuginfod for executable <%s> ... ", sbuild_id);
> > +
> > +       if (debuginfo) {
> > +               fd = debuginfod_find_debuginfo(c, (const unsigned char *) sbuild_id,
> > +                                              0, path);
> > +       } else {
> > +               fd = debuginfod_find_executable(c, (const unsigned char *) sbuild_id,
> > +                                               0, path);
> > +       }
> > +       if (fd >= 0)
> > +               close(fd); /* retaining reference by realname */
> > +
> > +       debuginfod_end(c);
> > +       pr_debug("%s%s\n", *path ? "OK " : "FAILED", *path ? *path : "");
> > +       return *path ? 0 : -1;
> > +}
> > +#else
> > +static int call_debuginfod(const char *sbuild_id __maybe_unused,
> > +                          char **path __maybe_unused,
> > +                          bool debuginfo __maybe_unused)
> > +{
> > +       return -1;
> > +}
> > +#endif
> > +
> > +struct dso_store_data {
> > +       bool     hits;
> > +       bool     force_download;
> 
> Where is it set?

it's not, I wanted to add it, but then never needed it,
so it stayed without an option.. I'll remove it

thanks,
jirka


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data
  2020-11-25 13:00   ` Namhyung Kim
  2020-11-25 13:04     ` Namhyung Kim
@ 2020-11-25 16:36     ` Jiri Olsa
  2020-11-26  4:49       ` Namhyung Kim
  1 sibling, 1 reply; 37+ messages in thread
From: Jiri Olsa @ 2020-11-25 16:36 UTC (permalink / raw)
  To: Namhyung Kim
  Cc: Jiri Olsa, Arnaldo Carvalho de Melo, lkml, Peter Zijlstra,
	Ingo Molnar, Mark Rutland, Alexander Shishkin, Michael Petlan,
	Song Liu, Ian Rogers, Stephane Eranian, Alexey Budankov,
	Andi Kleen, Adrian Hunter

On Wed, Nov 25, 2020 at 10:00:10PM +0900, Namhyung Kim wrote:
> On Tue, Nov 24, 2020 at 8:06 AM Jiri Olsa <jolsa@kernel.org> wrote:
> >
> > Adding support to specify perf data file as -a option file
> > argument,
> >
> > If the file is detected to be perf data file, it is processed
> > and all dso objects with sample hit are stored to the build
> > id cache.
> >
> >   $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data
> >   OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
> >   OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
> >
> > By default we store only dso with hits, but it's possible to
> > specify 'all' to store all dso objects, like:
> >     -a perf.data,all
> 
> I think we can add -A/--add-all like we have -p and -P.

hm, the thing is that 'all' is specific for perf data file: '-a perf.data'
hence -A 'file' would make no sense, only for '-A perf.data', so the current
'all' parameter seems less confusing to me

thanks,
jirka

> 
> >
> >   $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data,all
> >   OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
> >   OK   6ce92dc7c31f12fe5b7775a2bb8b14a3546ce2cd /lib/modules/5.10.0-rc2speed+/kernel/drivers/firmware/qemu_fw_cfg.ko
> >   OK   bf3f6d32dccc159f841fc3658c241d0e74c61fbb /lib/modules/5.10.0-rc2speed+/kernel/drivers/block/virtio_blk.ko
> >   OK   e896b4329cf9f190f1a0fae933f425ff8f71b052 /lib/modules/5.10.0-rc2speed+/kernel/drivers/char/virtio_console.ko
> >   OK   5bedc933cb59e053ecb472f327bd73c548364479 /lib/modules/5.10.0-rc2speed+/kernel/drivers/input/serio/serio_raw.ko
> >   OK   cecc506368a8b7a473a5f900d26f0d3d914a9c23 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32c-intel.ko
> >   OK   91076fb3646d061a0a42cf7bddb339a665ee4f80 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/ghash-clmulni-intel.ko
> >   OK   4e2a304d788bb8e2e950bc82a5944e042afa0bf2 /lib/modules/5.10.0-rc2speed+/kernel/drivers/media/cec/core/cec.ko
> >   OK   31ab0da5ad81e6803280177f507a95f3053d585e /lib/modules/5.10.0-rc2speed+/kernel/lib/libcrc32c.ko
> >   OK   f6154bca47c149f48c942fcc3d653041dd285c65 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/ttm/ttm.ko
> >   OK   723f5852de81590d54b23b38c160d3618b41951b /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crct10dif-pclmul.ko
> >   OK   06b1eab7f141cbc3e5a5db47909c8ab5cb242e40 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_ttm_helper.ko
> >   OK   38292b862cf3ff87489508fdb4895efa45780813 /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/qxl/qxl.ko
> >   OK   cdf51e58609bf2ce4837a7b195e0ccae0a930907 /lib/modules/5.10.0-rc2speed+/kernel/arch/x86/crypto/crc32-pclmul.ko
> >   OK   5ca8958388f6688452ecc2cb83d6031394c659ad /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm.ko
> >   OK   236bc4e4f38bf3559007566cb32b3dcc1bc28d2d /lib/modules/5.10.0-rc2speed+/kernel/drivers/gpu/drm/drm_kms_helper.ko
> >   OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
> >   OK   66db2be3efaa43bb5a5c481986e9554e1885cc69 /usr/lib/systemd/systemd
> >   OK   7db607d9f2de89860d9639712da64c8bacd31e4b /usr/lib64/libm-2.30.so
> >   OK   55b5f9652e1d17c1dd58f62628d5063428e5db91 /usr/lib64/libudev.so.1.6.15
> >   OK   63b97070bf097130713bb6c89cf7100b5f3c9b17 /usr/lib64/libunistring.so.2.1.0
> >   ...
> >
> > Once perf data is specified, no other file can be specified in
> > the option, otherwise it causes syntax error.
> >
> > Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> > ---
> >  .../perf/Documentation/perf-buildid-cache.txt |  12 +-
> >  tools/perf/builtin-buildid-cache.c            | 215 +++++++++++++++++-
> >  tools/perf/util/probe-event.c                 |   6 +-
> >  3 files changed, 227 insertions(+), 6 deletions(-)
> >
> > diff --git a/tools/perf/Documentation/perf-buildid-cache.txt b/tools/perf/Documentation/perf-buildid-cache.txt
> > index f6de0952ff3c..b77da5138bca 100644
> > --- a/tools/perf/Documentation/perf-buildid-cache.txt
> > +++ b/tools/perf/Documentation/perf-buildid-cache.txt
> > @@ -23,7 +23,17 @@ OPTIONS
> >  -------
> >  -a::
> >  --add=::
> > -        Add specified file to the cache.
> > +        Add specified file or perf.data binaries to the cache.
> > +
> > +        If the file is detected to be perf data file, it is processed
> > +        and all dso objects with sample hit are stored to the cache.
> > +
> > +        It's possible to specify 'all' to store all dso objects, like:
> > +            -a perf.data,all
> > +
> > +        Once perf data is specified, no other file can be specified in
> > +        the option, otherwise it causes syntax error.
> > +
> >  -f::
> >  --force::
> >         Don't complain, do it.
> > diff --git a/tools/perf/builtin-buildid-cache.c b/tools/perf/builtin-buildid-cache.c
> > index a25411926e48..0bfb54ee1e5e 100644
> > --- a/tools/perf/builtin-buildid-cache.c
> > +++ b/tools/perf/builtin-buildid-cache.c
> > @@ -29,6 +29,11 @@
> >  #include "util/probe-file.h"
> >  #include <linux/string.h>
> >  #include <linux/err.h>
> > +#include <linux/zalloc.h>
> > +#include <sys/stat.h>
> > +#ifdef HAVE_DEBUGINFOD_SUPPORT
> > +#include <elfutils/debuginfod.h>
> > +#endif
> >
> >  static int build_id_cache__kcore_buildid(const char *proc_dir, char *sbuildid)
> >  {
> > @@ -348,6 +353,205 @@ static int build_id_cache__show_all(void)
> >         return 0;
> >  }
> >
> > +#ifdef HAVE_DEBUGINFOD_SUPPORT
> 
> I think this part belongs to the next patch...?
> 
> Thanks,
> Namhyung
> 
> 
> > +static int call_debuginfod(const char *sbuild_id, char **path, bool debuginfo)
> > +{
> > +       debuginfod_client *c;
> > +       int fd;
> > +
> > +       c = debuginfod_begin();
> > +       if (c == NULL)
> > +               return -1;
> > +
> > +       pr_debug("trying debuginfod for executable <%s> ... ", sbuild_id);
> > +
> > +       if (debuginfo) {
> > +               fd = debuginfod_find_debuginfo(c, (const unsigned char *) sbuild_id,
> > +                                              0, path);
> > +       } else {
> > +               fd = debuginfod_find_executable(c, (const unsigned char *) sbuild_id,
> > +                                               0, path);
> > +       }
> > +       if (fd >= 0)
> > +               close(fd); /* retaining reference by realname */
> > +
> > +       debuginfod_end(c);
> > +       pr_debug("%s%s\n", *path ? "OK " : "FAILED", *path ? *path : "");
> > +       return *path ? 0 : -1;
> > +}
> > +#else
> > +static int call_debuginfod(const char *sbuild_id __maybe_unused,
> > +                          char **path __maybe_unused,
> > +                          bool debuginfo __maybe_unused)
> > +{
> > +       return -1;
> > +}
> > +#endif
> > +
> > +struct dso_store_data {
> > +       bool     hits;
> > +       bool     force_download;
> > +};
> > +
> > +static int dso_store(struct dso *dso, struct machine *machine, void *priv)
> > +{
> > +       struct dso_store_data *data = priv;
> > +       char sbuild_id[SBUILD_ID_SIZE];
> > +       struct build_id bid;
> > +       char *path = NULL, *link = NULL;
> > +       bool is_kallsyms;
> > +       int err = -1;
> > +
> > +       /*
> > +        * There's no build id in dso, nothing to do..
> > +        */
> > +       if (!dso->has_build_id || !build_id__is_defined(&dso->bid))
> > +               return 0;
> > +
> > +       if (data->hits && !dso->hit)
> > +               return 0;
> > +
> > +       /*
> > +        * The storing process is:
> > +        *   - get build id of the dso
> > +        *   - check if it is already in cache
> > +        *   - check if it matches provided build id from mmap2 event
> > +        *   - if not, try debuginfod to download the binary
> > +        *   - store binary to build id database
> > +        */
> > +       is_kallsyms = !strcmp(machine->mmap_name, dso->short_name);
> > +       build_id__sprintf(&dso->bid, sbuild_id);
> > +
> > +       link = build_id_cache__linkname(sbuild_id, NULL, 0);
> > +       if (!link)
> > +               return -ENOMEM;
> > +
> > +       if (!data->force_download && !access(link, X_OK)) {
> > +               pr_debug("already in cache - %s <%s>\n", dso->long_name, sbuild_id);
> > +               err = 0;
> > +               goto out;
> > +       }
> > +
> > +       path = strdup(dso->long_name);
> > +       if (!path)
> > +               goto out;
> > +
> > +       if (is_kallsyms) {
> > +               /*
> > +                * Find out if we are on the same kernel as perf.data
> > +                * and store kallsyms in that case.
> > +                */
> > +               err = sysfs__read_build_id("/sys/kernel/notes", &bid);
> > +               if (err < 0)
> > +                       goto out;
> > +       } else {
> > +               struct nscookie nsc;
> > +               struct stat st;
> > +
> > +               nsinfo__mountns_enter(dso->nsinfo, &nsc);
> > +
> > +               /*
> > +                * Does the file exists in the first place, if it does,
> > +                * resolve path and read the build id.
> > +                */
> > +               if (stat(dso->long_name, &st)) {
> > +                       nsinfo__mountns_exit(&nsc);
> > +                       zfree(&path);
> > +                       goto try_download;
> > +               }
> > +
> > +               err = filename__read_build_id(dso->long_name, &bid);
> > +               nsinfo__mountns_exit(&nsc);
> > +
> > +               if (err <= 0)
> > +                       goto out;
> > +       }
> > +
> > +       /*
> > +        * If we match, then what we want in mmap2 event
> > +        * is what we got in the binary,
> > +        */
> > +       if (bid.size != dso->bid.size || memcmp(&bid, &dso->bid, bid.size)) {
> > +               char sbid[SBUILD_ID_SIZE];
> > +
> > +               build_id__sprintf(&bid, sbid);
> > +               pr_debug("mmap build id <%s> does not match for %s <%s>\n",
> > +                        sbuild_id, path, sbid);
> > +               zfree(&path);
> > +       }
> > +
> > +try_download:
> > +       /*
> > +        * We did not match build id or did not find the
> > +        * binary - try debuginfod as last resort.
> > +        */
> > +       if (!path) {
> > +               bool debuginfo;
> > +               char *tmp = NULL;
> > +
> > +               /*
> > +                * The debuginfo retrieval for standard binaries
> > +                * is handled within build_id_cache__add function.
> > +                *
> > +                * For kernel and kernel modules we have to ask
> > +                * for debuginfo directly, because debuginfod
> > +                * does not treat them as binaries.
> > +                */
> > +               debuginfo = is_kallsyms ||
> > +                           is_kernel_module(dso->long_name, PERF_RECORD_MISC_CPUMODE_UNKNOWN);
> > +
> > +               if (call_debuginfod(sbuild_id, &tmp, debuginfo)) {
> > +                       err = -1;
> > +                       goto out;
> > +               }
> > +
> > +               path = tmp;
> > +
> > +               /*
> > +                * The kernel dso is now elf binary, so disable is_kallsyms
> > +                * so build_id_cache__add can prepare proper file names.
> > +                */
> > +               is_kallsyms = false;
> > +       }
> > +
> > +       pr_debug("linking %s %s <%s>\n", dso->short_name, path, sbuild_id);
> > +
> > +       err = build_id_cache__add(sbuild_id, path, path,
> > +                                 dso->nsinfo, is_kallsyms, false);
> > +out:
> > +       free(path);
> > +       fprintf(stderr, "%s %s %s\n", err ? "FAIL" : "OK  ", sbuild_id, dso->long_name);
> > +       return 0;
> > +}
> > +
> > +static int
> > +build_id_cache__add_perf_data(const char *path, bool all)
> > +{
> > +       struct perf_session *session;
> > +       struct dso_store_data priv = {
> > +               .hits = !all,
> > +               .force_download = false,
> > +       };
> > +       struct perf_data data = {
> > +               .path  = path,
> > +               .mode  = PERF_DATA_MODE_READ,
> > +       };
> > +       int err;
> > +
> > +       session = perf_session__new(&data, false, &build_id__mark_dso_hit_ops);
> > +       if (IS_ERR(session))
> > +               return PTR_ERR(session);
> > +
> > +       err = perf_session__process_events(session);
> > +       if (err)
> > +               goto out;
> > +
> > +       err = __perf_session__cache_build_ids(session, dso_store, &priv);
> > +out:
> > +       perf_session__delete(session);
> > +       return err;
> > +}
> > +
> >  int cmd_buildid_cache(int argc, const char **argv)
> >  {
> >         struct strlist *list;
> > @@ -440,7 +644,15 @@ int cmd_buildid_cache(int argc, const char **argv)
> >                 list = strlist__new(add_name_list_str, NULL);
> >                 if (list) {
> >                         strlist__for_each_entry(pos, list)
> > -                               if (build_id_cache__add_file(pos->s, nsi)) {
> > +                               if (is_perf_data(pos->s)) {
> > +                                       struct str_node *all_pos = strlist__next(pos);
> > +                                       bool all = !strcmp("all", all_pos ? all_pos->s : "");
> > +
> > +                                       if (build_id_cache__add_perf_data(pos->s, all))
> > +                                               pr_warning("Couldn't add build ids from %s\n", pos->s);
> > +                                       if (all)
> > +                                               pos = all_pos;
> > +                               } else if (build_id_cache__add_file(pos->s, nsi)) {
> >                                         if (errno == EEXIST) {
> >                                                 pr_debug("%s already in the cache\n",
> >                                                          pos->s);
> > @@ -449,7 +661,6 @@ int cmd_buildid_cache(int argc, const char **argv)
> >                                         pr_warning("Couldn't add %s: %s\n",
> >                                                    pos->s, str_error_r(errno, sbuf, sizeof(sbuf)));
> >                                 }
> > -
> >                         strlist__delete(list);
> >                 }
> >         }
> > diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
> > index 8eae2afff71a..e821bb977c9b 100644
> > --- a/tools/perf/util/probe-event.c
> > +++ b/tools/perf/util/probe-event.c
> > @@ -1616,9 +1616,9 @@ static int parse_perf_probe_point(char *arg, struct perf_probe_event *pev)
> >                 return -EINVAL;
> >         }
> >
> > -       pr_debug("symbol:%s file:%s line:%d offset:%lu return:%d lazy:%s\n",
> > -                pp->function, pp->file, pp->line, pp->offset, pp->retprobe,
> > -                pp->lazy_line);
> > +       pr_debug2("symbol:%s file:%s line:%d offset:%lu return:%d lazy:%s\n",
> > +                 pp->function, pp->file, pp->line, pp->offset, pp->retprobe,
> > +                 pp->lazy_line);
> >         return 0;
> >  }
> >
> > --
> > 2.26.2
> >
> 


^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data
  2020-11-25 16:36     ` Jiri Olsa
@ 2020-11-26  4:49       ` Namhyung Kim
  0 siblings, 0 replies; 37+ messages in thread
From: Namhyung Kim @ 2020-11-26  4:49 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: Jiri Olsa, Arnaldo Carvalho de Melo, lkml, Peter Zijlstra,
	Ingo Molnar, Mark Rutland, Alexander Shishkin, Michael Petlan,
	Song Liu, Ian Rogers, Stephane Eranian, Alexey Budankov,
	Andi Kleen, Adrian Hunter

On Thu, Nov 26, 2020 at 1:36 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Wed, Nov 25, 2020 at 10:00:10PM +0900, Namhyung Kim wrote:
> > On Tue, Nov 24, 2020 at 8:06 AM Jiri Olsa <jolsa@kernel.org> wrote:
> > >
> > > Adding support to specify perf data file as -a option file
> > > argument,
> > >
> > > If the file is detected to be perf data file, it is processed
> > > and all dso objects with sample hit are stored to the build
> > > id cache.
> > >
> > >   $ DEBUGINFOD_URLS=http://192.168.122.174:8002 perf buildid-cache -a perf.data
> > >   OK   5dcec522abf136fcfd3128f47e131f2365834dd7 /home/jolsa/.debug/.build-id/5d/cec522abf136fcfd3128f47e131f2365834dd7/elf
> > >   OK   5784f813b727a50cfd3363234aef9fcbab685cc4 /lib/modules/5.10.0-rc2speed+/kernel/fs/xfs/xfs.ko
> > >
> > > By default we store only dso with hits, but it's possible to
> > > specify 'all' to store all dso objects, like:
> > >     -a perf.data,all
> >
> > I think we can add -A/--add-all like we have -p and -P.
>
> hm, the thing is that 'all' is specific for perf data file: '-a perf.data'
> hence -A 'file' would make no sense, only for '-A perf.data', so the current
> 'all' parameter seems less confusing to me

Yeah, I also thought about the '-A perf.data' form.
But I won't insist on it strongly, it's up to you. :)

Thanks,
Namhyung

^ permalink raw reply	[flat|nested] 37+ messages in thread

* Re: [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids
  2020-11-26 17:00 ` [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids Jiri Olsa
@ 2020-11-26 17:52   ` Arnaldo Carvalho de Melo
  0 siblings, 0 replies; 37+ messages in thread
From: Arnaldo Carvalho de Melo @ 2020-11-26 17:52 UTC (permalink / raw)
  To: Jiri Olsa
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Em Thu, Nov 26, 2020 at 06:00:19PM +0100, Jiri Olsa escreveu:
> Using machine__for_each_dso in perf_session__cache_build_ids,
> so we can reuse perf_session__cache_build_ids with different
> callback in following changes.

Thanks, applied.

- Arnaldo

 
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
>  tools/perf/util/build-id.c | 41 +++++++++++++++-----------------------
>  1 file changed, 16 insertions(+), 25 deletions(-)
> 
> diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
> index 1fd58703c2d4..948a7f48d668 100644
> --- a/tools/perf/util/build-id.c
> +++ b/tools/perf/util/build-id.c
> @@ -859,12 +859,16 @@ int build_id_cache__remove_s(const char *sbuild_id)
>  	return err;
>  }
>  
> -static int dso__cache_build_id(struct dso *dso, struct machine *machine)
> +static int dso__cache_build_id(struct dso *dso, struct machine *machine,
> +			       void *priv __maybe_unused)
>  {
>  	bool is_kallsyms = dso__is_kallsyms(dso);
>  	bool is_vdso = dso__is_vdso(dso);
>  	const char *name = dso->long_name;
>  
> +	if (!dso->has_build_id)
> +		return 0;
> +
>  	if (dso__is_kcore(dso)) {
>  		is_kallsyms = true;
>  		name = machine->mmap_name;
> @@ -873,43 +877,30 @@ static int dso__cache_build_id(struct dso *dso, struct machine *machine)
>  				     is_kallsyms, is_vdso);
>  }
>  
> -static int __dsos__cache_build_ids(struct list_head *head,
> -				   struct machine *machine)
> +static int
> +machines__for_each_dso(struct machines *machines, machine__dso_t fn, void *priv)
>  {
> -	struct dso *pos;
> -	int err = 0;
> -
> -	dsos__for_each_with_build_id(pos, head)
> -		if (dso__cache_build_id(pos, machine))
> -			err = -1;
> +	int ret = machine__for_each_dso(&machines->host, fn, priv);
> +	struct rb_node *nd;
>  
> -	return err;
> -}
> +	for (nd = rb_first_cached(&machines->guests); nd;
> +	     nd = rb_next(nd)) {
> +		struct machine *pos = rb_entry(nd, struct machine, rb_node);
>  
> -static int machine__cache_build_ids(struct machine *machine)
> -{
> -	return __dsos__cache_build_ids(&machine->dsos.head, machine);
> +		ret |= machine__for_each_dso(pos, fn, priv);
> +	}
> +	return ret ? -1 : 0;
>  }
>  
>  int perf_session__cache_build_ids(struct perf_session *session)
>  {
> -	struct rb_node *nd;
> -	int ret;
> -
>  	if (no_buildid_cache)
>  		return 0;
>  
>  	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
>  		return -1;
>  
> -	ret = machine__cache_build_ids(&session->machines.host);
> -
> -	for (nd = rb_first_cached(&session->machines.guests); nd;
> -	     nd = rb_next(nd)) {
> -		struct machine *pos = rb_entry(nd, struct machine, rb_node);
> -		ret |= machine__cache_build_ids(pos);
> -	}
> -	return ret ? -1 : 0;
> +	return machines__for_each_dso(&session->machines, dso__cache_build_id, NULL) ?  -1 : 0;
>  }
>  
>  static bool machine__read_build_ids(struct machine *machine, bool with_hits)
> -- 
> 2.26.2
> 

-- 

- Arnaldo

^ permalink raw reply	[flat|nested] 37+ messages in thread

* [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids
  2020-11-26 17:00 [PATCHv4 " Jiri Olsa
@ 2020-11-26 17:00 ` Jiri Olsa
  2020-11-26 17:52   ` Arnaldo Carvalho de Melo
  0 siblings, 1 reply; 37+ messages in thread
From: Jiri Olsa @ 2020-11-26 17:00 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo
  Cc: lkml, Peter Zijlstra, Ingo Molnar, Mark Rutland, Namhyung Kim,
	Alexander Shishkin, Michael Petlan, Song Liu, Ian Rogers,
	Stephane Eranian, Alexey Budankov, Andi Kleen, Adrian Hunter

Using machine__for_each_dso in perf_session__cache_build_ids,
so we can reuse perf_session__cache_build_ids with different
callback in following changes.

Signed-off-by: Jiri Olsa <jolsa@kernel.org>
---
 tools/perf/util/build-id.c | 41 +++++++++++++++-----------------------
 1 file changed, 16 insertions(+), 25 deletions(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 1fd58703c2d4..948a7f48d668 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -859,12 +859,16 @@ int build_id_cache__remove_s(const char *sbuild_id)
 	return err;
 }
 
-static int dso__cache_build_id(struct dso *dso, struct machine *machine)
+static int dso__cache_build_id(struct dso *dso, struct machine *machine,
+			       void *priv __maybe_unused)
 {
 	bool is_kallsyms = dso__is_kallsyms(dso);
 	bool is_vdso = dso__is_vdso(dso);
 	const char *name = dso->long_name;
 
+	if (!dso->has_build_id)
+		return 0;
+
 	if (dso__is_kcore(dso)) {
 		is_kallsyms = true;
 		name = machine->mmap_name;
@@ -873,43 +877,30 @@ static int dso__cache_build_id(struct dso *dso, struct machine *machine)
 				     is_kallsyms, is_vdso);
 }
 
-static int __dsos__cache_build_ids(struct list_head *head,
-				   struct machine *machine)
+static int
+machines__for_each_dso(struct machines *machines, machine__dso_t fn, void *priv)
 {
-	struct dso *pos;
-	int err = 0;
-
-	dsos__for_each_with_build_id(pos, head)
-		if (dso__cache_build_id(pos, machine))
-			err = -1;
+	int ret = machine__for_each_dso(&machines->host, fn, priv);
+	struct rb_node *nd;
 
-	return err;
-}
+	for (nd = rb_first_cached(&machines->guests); nd;
+	     nd = rb_next(nd)) {
+		struct machine *pos = rb_entry(nd, struct machine, rb_node);
 
-static int machine__cache_build_ids(struct machine *machine)
-{
-	return __dsos__cache_build_ids(&machine->dsos.head, machine);
+		ret |= machine__for_each_dso(pos, fn, priv);
+	}
+	return ret ? -1 : 0;
 }
 
 int perf_session__cache_build_ids(struct perf_session *session)
 {
-	struct rb_node *nd;
-	int ret;
-
 	if (no_buildid_cache)
 		return 0;
 
 	if (mkdir(buildid_dir, 0755) != 0 && errno != EEXIST)
 		return -1;
 
-	ret = machine__cache_build_ids(&session->machines.host);
-
-	for (nd = rb_first_cached(&session->machines.guests); nd;
-	     nd = rb_next(nd)) {
-		struct machine *pos = rb_entry(nd, struct machine, rb_node);
-		ret |= machine__cache_build_ids(pos);
-	}
-	return ret ? -1 : 0;
+	return machines__for_each_dso(&session->machines, dso__cache_build_id, NULL) ?  -1 : 0;
 }
 
 static bool machine__read_build_ids(struct machine *machine, bool with_hits)
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 37+ messages in thread

end of thread, other threads:[~2020-11-26 17:52 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-23 23:04 [PATCHv3 00/25] perf: Add mmap2 build id support Jiri Olsa
2020-11-23 23:04 ` [PATCH 01/25] bpf: Move stack_map_get_build_id into lib Jiri Olsa
2020-11-23 23:04 ` [PATCH 02/25] bpf: Add size arg to build_id_parse function Jiri Olsa
2020-11-23 23:04 ` [PATCH 03/25] perf: Add build id data in mmap2 event Jiri Olsa
2020-11-23 23:04 ` [PATCH 04/25] tools headers uapi: Sync tools/include/uapi/linux/perf_event.h Jiri Olsa
2020-11-23 23:04 ` [PATCH 05/25] tools lib: Adopt memchr_inv() from kernel Jiri Olsa
2020-11-23 23:04 ` [PATCH 06/25] perf tools: Do not swap mmap2 fields in case it contains build id Jiri Olsa
2020-11-23 23:04 ` [PATCH 07/25] perf tools: Add build_id__is_defined function Jiri Olsa
2020-11-23 23:04 ` [PATCH 08/25] perf tools: Add filename__decompress function Jiri Olsa
2020-11-23 23:04 ` [PATCH 09/25] perf tools: Add support to read build id from compressed elf Jiri Olsa
2020-11-23 23:04 ` [PATCH 10/25] perf tools: Add check for existing link in buildid dir Jiri Olsa
2020-11-23 23:04 ` [PATCH 11/25] perf tools: Use struct extra_kernel_map in machine__process_kernel_mmap_event Jiri Olsa
2020-11-23 23:04 ` [PATCH 12/25] perf tools: Try to load vmlinux from buildid database Jiri Olsa
2020-11-23 23:05 ` [PATCH 13/25] perf tools: Store build id from mmap2 events Jiri Olsa
2020-11-25 12:56   ` Namhyung Kim
2020-11-25 14:15     ` Jiri Olsa
2020-11-23 23:05 ` [PATCH 14/25] perf tools: Allow mmap2 event to synthesize kernel image Jiri Olsa
2020-11-23 23:05 ` [PATCH 15/25] perf tools: Allow mmap2 event to synthesize modules Jiri Olsa
2020-11-23 23:05 ` [PATCH 16/25] perf tools: Synthesize build id for kernel/modules/tasks Jiri Olsa
2020-11-23 23:05 ` [PATCH 17/25] perf tools: Add support to display build id for mmap2 events Jiri Olsa
2020-11-23 23:05 ` [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids Jiri Olsa
2020-11-23 23:05 ` [PATCH 19/25] perf tools: Add __perf_session__cache_build_ids function Jiri Olsa
2020-11-23 23:05 ` [PATCH 20/25] perf tools: Add is_perf_data function Jiri Olsa
2020-11-23 23:05 ` [PATCH 21/25] perf tools: Add build_id_cache__add function Jiri Olsa
2020-11-23 23:05 ` [PATCH 22/25] perf buildid-cache: Add support to add build ids from perf data Jiri Olsa
2020-11-25 13:00   ` Namhyung Kim
2020-11-25 13:04     ` Namhyung Kim
2020-11-25 16:36     ` Jiri Olsa
2020-11-26  4:49       ` Namhyung Kim
2020-11-25 13:29   ` Namhyung Kim
2020-11-25 14:20     ` Jiri Olsa
2020-11-23 23:05 ` [PATCH 23/25] perf buildid-cache: Add --debuginfod option Jiri Olsa
2020-11-23 23:05 ` [PATCH 24/25] perf buildid-list: Add support for mmap2's buildid events Jiri Olsa
2020-11-23 23:05 ` [PATCH 25/25] perf record: Add --buildid-mmap option to enable mmap's build id Jiri Olsa
2020-11-25 12:54 ` [PATCHv3 00/25] perf: Add mmap2 build id support Namhyung Kim
2020-11-26 17:00 [PATCHv4 " Jiri Olsa
2020-11-26 17:00 ` [PATCH 18/25] perf tools: Use machine__for_each_dso in perf_session__cache_build_ids Jiri Olsa
2020-11-26 17:52   ` Arnaldo Carvalho de Melo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).