From: Barry Song <21cnbao@gmail.com>
To: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>, Qi Liu <liuqi115@huawei.com>,
Catalin Marinas <catalin.marinas@arm.com>,
will@kernel.org, naveen.n.rao@linux.ibm.com,
anil.s.keshavamurthy@intel.com,
David Miller <davem@davemloft.net>,
linux-arm-kernel@lists.infradead.org,
Barry Song <song.bao.hua@hisilicon.com>,
prime.zeng@hisilicon.com, robin.murphy@arm.com,
f.fangjian@huawei.com, Linuxarm <linuxarm@huawei.com>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v4 2/2] arm64: kprobe: Enable OPTPROBE for arm64
Date: Wed, 25 Aug 2021 15:12:04 +1200 [thread overview]
Message-ID: <CAGsJ_4ww6wLBHw++qE_29ozC50XE8qc4T5waxn78UHRHjUFunQ@mail.gmail.com> (raw)
In-Reply-To: <20210825111339.dcf494abb0c27508d2d1f645@kernel.org>
On Wed, Aug 25, 2021 at 2:15 PM Masami Hiramatsu <mhiramat@kernel.org> wrote:
>
> On Tue, 24 Aug 2021 11:50:01 +0100
> Mark Rutland <mark.rutland@arm.com> wrote:
>
> > Hi,
> >
> > I have a bunch of comments below.
> >
> > At a high-level, I'm not all that keen on adding yet another set of
> > trampolines, especially given we have constraints on how we can branch
> > to them which render this not that useful in common configurations (e.g.
> > where KASLR and module randomization is enabled).
>
> Yes, that makes kprobe jump optimization hard to implement on
> RISC architecture in general. (x86 has 32bit offset jump instruction)
> To solve this issue, something like "intermedate jump area" is needed
> for each module. (Or, overwriting multiple instructions)
>
> >
> > So importantly, do we actually need this? I don't think the sampel is
> > that compelling since we can already use ftrace to measure function
> > latencies.
>
> That depends on what you use it for, as you may know, kprobes allows
> you to put the probes on function body (and inlined function),
> on the other hand, ftrace can put only on the entry of the function.
> I guess Qi may want to use it for improving performance of BPF.
>
> (BTW, as far as I know, Jisheng Zhang once tried to implement
> kprobe on ftrace, that may be more helpful in this example.
> https://lore.kernel.org/linux-arm-kernel/20191225172625.69811b3e@xhacker.debian/T/#m23a7aa55d32d140ee6a92102534446cfd4a43007
> I will pick them up again)
>
> >
> > If we do need this, I think we need to do some more substantial rework
> > to address those branch range limitations. I know that we could permit
> > arbitrary branching if we expand the ftrace-with-regs callsites to ~6
> > instructions, but that interacts rather poorly with stacktracing and
> > will make the kernel a bit bigger.
>
> Would you mean we reuse the ftrace-with-regs callsites for kprobes?
>
> arm32 avoids this limitation partially with reserved text pages
> for trampoline in the kernel. But I think that is also a partial
> solution. It may not work with module randomization at least on
> arm64.
>
> On arm64, I think there are several way to solve it.
>
> - Add optprobe trampoline buffer for each module.
> This is the simplest way to solve this issue, but requires some
> pages to be added to each module (and kernel).
>
> - Add intermediate trampoline area for each module. (2-stage jump)
> This jumps into an intermediate trampoline entry, save a partial
> registers and jump the actual trampoline using that register.
> This can reduce the size of trampoline buffer for each module.
>
> - Replace multiple instructions with the above intermediate jump
> code. (single jump, but replace multiple instructions)
> This requires to emulate multiple instructions and also the
> kprobe must decode the instructions in the target function to
> identify the replaced instructions are in one basic block. But
> no need to add intermediate trampoline area (page).
>
>
>
> >
> > On Wed, Aug 18, 2021 at 03:33:36PM +0800, Qi Liu wrote:
> > > This patch introduce optprobe for ARM64. In optprobe, probed
> > > instruction is replaced by a branch instruction to detour
> > > buffer. Detour buffer contains trampoline code and a call to
> > > optimized_callback(). optimized_callback() calls opt_pre_handler()
> > > to execute kprobe handler.
> > >
> > > Performance of optprobe on Hip08 platform is test using kprobe
> > > example module[1] to analyze the latency of a kernel function,
> > > and here is the result:
> > >
> > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/kprobes/kretprobe_example.c
> > >
> > > kprobe before optimized:
> > > [280709.846380] do_empty returned 0 and took 1530 ns to execute
> > > [280709.852057] do_empty returned 0 and took 550 ns to execute
> > > [280709.857631] do_empty returned 0 and took 440 ns to execute
> > > [280709.863215] do_empty returned 0 and took 380 ns to execute
> > > [280709.868787] do_empty returned 0 and took 360 ns to execute
> > > [280709.874362] do_empty returned 0 and took 340 ns to execute
> > > [280709.879936] do_empty returned 0 and took 320 ns to execute
> > > [280709.885505] do_empty returned 0 and took 300 ns to execute
> > > [280709.891075] do_empty returned 0 and took 280 ns to execute
> > > [280709.896646] do_empty returned 0 and took 290 ns to execute
> > > [280709.902220] do_empty returned 0 and took 290 ns to execute
> > > [280709.907807] do_empty returned 0 and took 290 ns to execute
> > >
> > > optprobe:
> > > [ 2965.964572] do_empty returned 0 and took 90 ns to execute
> > > [ 2965.969952] do_empty returned 0 and took 80 ns to execute
> > > [ 2965.975332] do_empty returned 0 and took 70 ns to execute
> > > [ 2965.980714] do_empty returned 0 and took 60 ns to execute
> > > [ 2965.986128] do_empty returned 0 and took 80 ns to execute
> > > [ 2965.991507] do_empty returned 0 and took 70 ns to execute
> > > [ 2965.996884] do_empty returned 0 and took 70 ns to execute
> > > [ 2966.002262] do_empty returned 0 and took 80 ns to execute
> > > [ 2966.007642] do_empty returned 0 and took 70 ns to execute
> > > [ 2966.013020] do_empty returned 0 and took 70 ns to execute
> > > [ 2966.018400] do_empty returned 0 and took 70 ns to execute
> > > [ 2966.023779] do_empty returned 0 and took 70 ns to execute
> > > [ 2966.029158] do_empty returned 0 and took 70 ns to execute
> >
> > Do we have any examples of where this latency matters in practice?
> >
> > >
> > > Signed-off-by: Qi Liu <liuqi115@huawei.com>
> > >
> > > Note:
> > > To guarantee the offset between probe point and kprobe pre_handler
> > > is smaller than 128MiB, users should set
> > > CONFIG_RANDOMIZE_MODULE_REGION_FULL=N or set nokaslr in command line, or
> > > optprobe will not work and fall back to normal kprobe.
> >
> > Hmm... I don't think that's something we want to recommend, and
> > certainly distros *should* use KASLR and
> > CONFIG_RANDOMIZE_MODULE_REGION_FULL.
> >
> > What happens with defconfig? Do we always get the fallback behaviour?
>
> Yes, in such case, it fails back to normal kprobe.
just one minor comment. as Qi pointed out before, bootargs nokaslr
will make kernel built by defconfig
use optprobe:
nokaslr [KNL]
When CONFIG_RANDOMIZE_BASE is set, this disables
kernel and module base offset ASLR (Address Space
Layout Randomization).
in lab, while security is not a concern as online, it would be a good option.
> Anyway, optprobe is a background optimization. User can not specify
> which kprobe is optimized. That is automatically done.
>
> Thank you,
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>
Thanks
barry
next prev parent reply other threads:[~2021-08-25 3:12 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-18 7:33 [PATCH v4 0/2] arm64: Enable OPTPROBE for arm64 Qi Liu
2021-08-18 7:33 ` [PATCH v4 1/2] Make save_all_base_regs and restore_all_base_regs as common macro Qi Liu
2021-08-18 7:33 ` [PATCH v4 2/2] arm64: kprobe: Enable OPTPROBE for arm64 Qi Liu
2021-08-18 16:27 ` Masami Hiramatsu
2021-08-24 10:50 ` Mark Rutland
2021-08-24 11:50 ` Barry Song
2021-08-24 12:11 ` Mark Rutland
2021-08-24 12:42 ` Barry Song
2021-08-25 2:13 ` Masami Hiramatsu
2021-08-25 3:12 ` Barry Song [this message]
2021-09-07 3:14 ` liuqi (BA)
2021-11-26 10:31 ` liuqi (BA)
2021-11-27 12:23 ` Masami Hiramatsu
2021-11-29 1:40 ` liuqi (BA)
2021-11-29 5:00 ` Masami Hiramatsu
2021-11-29 6:50 ` liuqi (BA)
2021-11-29 14:35 ` Masami Hiramatsu
2021-11-30 6:48 ` liuqi (BA)
2021-12-01 1:50 ` Masami Hiramatsu
2021-12-01 2:55 ` liuqi (BA)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAGsJ_4ww6wLBHw++qE_29ozC50XE8qc4T5waxn78UHRHjUFunQ@mail.gmail.com \
--to=21cnbao@gmail.com \
--cc=anil.s.keshavamurthy@intel.com \
--cc=catalin.marinas@arm.com \
--cc=davem@davemloft.net \
--cc=f.fangjian@huawei.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxarm@huawei.com \
--cc=liuqi115@huawei.com \
--cc=mark.rutland@arm.com \
--cc=mhiramat@kernel.org \
--cc=naveen.n.rao@linux.ibm.com \
--cc=prime.zeng@hisilicon.com \
--cc=robin.murphy@arm.com \
--cc=song.bao.hua@hisilicon.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).