From: Masami Hiramatsu <mhiramat@kernel.org> To: Evgenii Shatokhin <eshatokhin@virtuozzo.com> Cc: Arnaldo Carvalho de Melo <acme@kernel.org>, Kristen Carlson Accardi <kristen@linux.intel.com>, live-patching@vger.kernel.org, Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@redhat.com>, linux-kernel@vger.kernel.org, Konstantin Khorenko <khorenko@virtuozzo.com> Subject: Re: 'perf probe' and symbols from .text.<something> Date: Tue, 23 Feb 2021 00:05:08 +0900 [thread overview] Message-ID: <20210223000508.cab3cddaa3a3790525f49247@kernel.org> (raw) In-Reply-To: <09257fb8-3ded-07b0-b3cc-55d5431698d8@virtuozzo.com> Hi Evgenii, On Thu, 18 Feb 2021 20:09:17 +0300 Evgenii Shatokhin <eshatokhin@virtuozzo.com> wrote: > Hi, > > It seems, 'perf probe' can only see functions from .text section in the > kernel modules, but not from .text.unlikely or other .text.* sections. > > For example, with kernel 5.11 and nf_conntrack.ko with debug info, 'perf > probe' succeeds for nf_conntrack_attach() from .text and fails for > nf_ct_resolve_clash() from .text.unlikely: Thanks for reporting it! > > ------------ > # perf probe -v -m nf_conntrack nf_ct_resolve_clash > probe-definition(0): nf_ct_resolve_clash > symbol:nf_ct_resolve_clash file:(null) line:0 offset:0 return:0 lazy:(null) > 0 arguments > Failed to get build-id from nf_conntrack. > Cache open error: -1 > Open Debuginfo file: > /lib/modules/5.11.0-test01/kernel/net/netfilter/nf_conntrack.ko > Try to find probe point from debuginfo. > Matched function: nf_ct_resolve_clash [33616] > Probe point found: nf_ct_resolve_clash+0 > Found 1 probe_trace_events. > Post processing failed or all events are skipped. (-2) > Probe point 'nf_ct_resolve_clash' not found. > Error: Failed to add events. Reason: No such file or directory (Code: -2) The above log shows, an error occured while post_process_probe_trace_events(), and the error code is -ENOENT (-2). ---- pr_debug("Found %d probe_trace_events.\n", ntevs); ret = post_process_probe_trace_events(pev, *tevs, ntevs, pev->target, pev->uprobes, dinfo); if (ret < 0 || ret == ntevs) { pr_debug("Post processing failed or all events are skipped. (%d)\n", ret); ---- In that function, map__find_symbol() failure will return -ENOENT. ---- /* Adjust symbol name and address */ static int post_process_probe_trace_point(struct probe_trace_point *tp, struct map *map, unsigned long offs) { struct symbol *sym; u64 addr = tp->address - offs; sym = map__find_symbol(map, addr); if (!sym) return -ENOENT; ---- So it seems "map" may not load the symbol out of ".text". This need to be fixed, since the map is widely used in the perf. Anyway, since this is on a module, so even if it can not find the symbol from map (or failed to load a map), it can fail back to the original symbol. Let me fix that. > # perf probe -v -m nf_conntrack nf_conntrack_attach > probe-definition(0): nf_conntrack_attach > symbol:nf_conntrack_attach file:(null) line:0 offset:0 return:0 lazy:(null) > 0 arguments > Failed to get build-id from nf_conntrack. > Cache open error: -1 > Open Debuginfo file: > /lib/modules/5.11.0-test01/kernel/net/netfilter/nf_conntrack.ko > Try to find probe point from debuginfo. > Matched function: nf_conntrack_attach [2c8c3] > Probe point found: nf_conntrack_attach+0 > Found 1 probe_trace_events. > Opening /sys/kernel/tracing//kprobe_events write=1 > Opening /sys/kernel/tracing//README write=0 > Writing event: p:probe/nf_conntrack_attach > nf_conntrack:nf_conntrack_attach+0 > Added new event: > probe:nf_conntrack_attach (on nf_conntrack_attach in nf_conntrack) > ------------ > > Is there a way to allow probing of functions in .text.<something> ? I need to check how machine__kernel_maps() generated maps cut down .text.unlikely. Arnaldo, I thought the maps in machine__kernel_maps() are generated from kallsyms (doesn't check .text) right? > > Of course, one could place probes using absolute addresses of the > functions but that would be less convenient. > > This also affects many livepatch modules where the kernel code can be > compiled with -ffunction-sections and each function may end up in a > separate section .text.<function_name>. 'perf probe' cannot be used > there, except with the absolute addresses. > > Moreover, if FGKASLR patches are merged > (https://lwn.net/Articles/832434/) and the kernel is built with FGKASLR > enabled, -ffunction-sections will be used too. 'perf probe' will be > unable to see the kernel functions then. Hmm, if the FGKASLAR really randomizes the symbol address, perf-probe should give up "_text-relative" probe for that kernel, and must fallback to the "symbol-based" probe. (Are there any way to check the FGKASLR is on?) The problem of "symbol-based" probe is that local (static) symbols may share a same name sometimes. In that case, it can not find correct symbol. (Maybe I can find a candidate from its size.) Anyway, sometimes the security and usability are trade-off. Thank you, -- Masami Hiramatsu <mhiramat@kernel.org>
next prev parent reply other threads:[~2021-02-22 15:06 UTC|newest] Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-02-18 17:09 Evgenii Shatokhin 2021-02-18 19:35 ` Josh Poimboeuf 2021-02-22 15:05 ` Masami Hiramatsu [this message] 2021-02-22 15:12 ` Masami Hiramatsu 2021-02-22 17:51 ` Josh Poimboeuf 2021-02-23 1:23 ` Masami Hiramatsu 2021-02-23 7:36 ` Masami Hiramatsu 2021-02-23 19:45 ` Josh Poimboeuf 2021-02-24 8:00 ` Masami Hiramatsu 2021-02-23 1:48 ` [PATCH] perf-probe: Failback to symbol-base probe for probes on module Masami Hiramatsu 2021-02-23 7:36 ` Masami Hiramatsu 2021-02-23 7:09 ` 'perf probe' and symbols from .text.<something> Masami Hiramatsu 2021-02-23 7:37 ` [PATCH] perf-probe: dso: Add symbols in .text.* subsections to text symbol map in kenrel modules Masami Hiramatsu 2021-02-23 15:02 ` Evgenii Shatokhin 2021-02-23 20:11 ` Arnaldo Carvalho de Melo 2021-02-24 7:47 ` Evgenii Shatokhin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210223000508.cab3cddaa3a3790525f49247@kernel.org \ --to=mhiramat@kernel.org \ --cc=acme@kernel.org \ --cc=eshatokhin@virtuozzo.com \ --cc=khorenko@virtuozzo.com \ --cc=kristen@linux.intel.com \ --cc=linux-kernel@vger.kernel.org \ --cc=live-patching@vger.kernel.org \ --cc=mingo@redhat.com \ --cc=peterz@infradead.org \ --subject='Re: '\''perf probe'\'' and symbols from .text.<something>' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).