From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65E76C4338F for ; Wed, 25 Aug 2021 03:14:24 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 26AEA61101 for ; Wed, 25 Aug 2021 03:14:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 26AEA61101 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xoqYhNOaEE/ESBrzbz5wVMpmKMmQYZ4i5ts32yHX5OQ=; b=4ixN/O2eqtVgm/ NFZwtqu4TjhTfyD0h2iar6uShMdBZXLFDUg83rsa15DHQtkbRhGeP5y+LR4obdlpjeilGzp3jJUSP gw4WuqzLCktZYwkfVKenkvP9zoUHuA6umXQCxyPx65NClAtKKwpxfHia1tRddreEPwqaPxuzBBVH/ TAVYdKn6NP6reseG3iaNSt3okZbTFvIyQD4wx18YiBQLKDclJU7nBKLPdFTnMdTfEsnvhlToVQrH4 M0fxM6ZsvOUT/YvqTTs3VMtu7qA7RoYdQh4t0L6sWGi5NrgNr4EDqep/rafpZKPFsa8MUYG20NSwB /N+fYtR508hTJ4kIUjTw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mIjKu-005Ki5-JV; Wed, 25 Aug 2021 03:12:20 +0000 Received: from mail-ot1-x329.google.com ([2607:f8b0:4864:20::329]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mIjKq-005Khj-Gg for linux-arm-kernel@lists.infradead.org; Wed, 25 Aug 2021 03:12:18 +0000 Received: by mail-ot1-x329.google.com with SMTP id o16-20020a9d2210000000b0051b1e56c98fso37002855ota.8 for ; Tue, 24 Aug 2021 20:12:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Uwyl1LEbZl8NmH1etx1NtKcHu409L3wd9+kEi6jB58o=; b=Pxma9Q/wrDG3/Bop4MDR3dft/B83gpxAMNvJ2QaFzJaVw2cCagIgH5mDk5hW660Dbm ViCsIT1kBwt3pyI/0Ds4p76S2iJuMFu8DLDYiN/a0l/s3Ty+jk4v/4431Fw51ABuldoz EtLx2BhIPKizrOOr+D9LD7Wmsn1kq/QHjyaXT0AgzaDw9dzQLFEnBI5je4ZaJlZD8rqQ u4nUWLIwNH+5ss49crI6Fav76KsXovua4CH+2FsEWg5puktu+qepyxd+pK3fkiFIfkaE BoDjcpDIk9h6yxEEbZrWOJvan+5Ynl6X4ZwaTZxEQwfBMzb+EqierB7tEW1QZKni/nt4 n4Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Uwyl1LEbZl8NmH1etx1NtKcHu409L3wd9+kEi6jB58o=; b=nLOwYhs9hO7OtULSnNb3biUeWJrJEvJbKweUw2VOAkyXQKQG7rfS0AvFNDavyDgCD8 KsqItS9ppt+fettUZ11jbgNIuIpVChkreNwe9AzDhlJ+gvrHsaMEzNefCanwkROXYo1S KbMPFenDvks6EA76vfplWajTSmK24q0xGpXSZD3se3m2+Y9fWIV4AkqutAn6k+srrz0N x1KB1oOekwsxbLJGl+gNm5zfusYEKsTQMLiUYbEXQUsvnE0vUEUO3Ekaxypq1UVw/AhF /poVPjN6tz2zgwD7Gh4QNPNQQ9HSsrkj/vFTYY/GZQRDy2bQd/12yBpjOtMzl6oCqRIg WuOw== X-Gm-Message-State: AOAM5317OTx+ecZIIEFJ1+5QvXclNwMZ/pwqEXQlBfYqlTh74Bm4IN5z L9e3eEt/Mi1j1xGX8ncsi8qKPOEL3iPR8sy2C/0= X-Google-Smtp-Source: ABdhPJyP6pnXPYR36sZqIr4FLuqHbZv/icnmGxuDycdfnGsMWGYh20jOi5PFJcKofIGVHSAcYQrW+HKEECrmOWakeTs= X-Received: by 2002:aca:220a:: with SMTP id b10mr5268581oic.101.1629861135614; Tue, 24 Aug 2021 20:12:15 -0700 (PDT) MIME-Version: 1.0 References: <20210818073336.59678-1-liuqi115@huawei.com> <20210818073336.59678-3-liuqi115@huawei.com> <20210824105001.GA96738@C02TD0UTHF1T.local> <20210825111339.dcf494abb0c27508d2d1f645@kernel.org> In-Reply-To: <20210825111339.dcf494abb0c27508d2d1f645@kernel.org> From: Barry Song <21cnbao@gmail.com> Date: Wed, 25 Aug 2021 15:12:04 +1200 Message-ID: Subject: Re: [PATCH v4 2/2] arm64: kprobe: Enable OPTPROBE for arm64 To: Masami Hiramatsu Cc: Mark Rutland , Qi Liu , Catalin Marinas , will@kernel.org, naveen.n.rao@linux.ibm.com, anil.s.keshavamurthy@intel.com, David Miller , linux-arm-kernel@lists.infradead.org, Barry Song , prime.zeng@hisilicon.com, robin.murphy@arm.com, f.fangjian@huawei.com, Linuxarm , LKML X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210824_201216_605021_257D6A03 X-CRM114-Status: GOOD ( 47.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 25, 2021 at 2:15 PM Masami Hiramatsu wrote: > > On Tue, 24 Aug 2021 11:50:01 +0100 > Mark Rutland wrote: > > > Hi, > > > > I have a bunch of comments below. > > > > At a high-level, I'm not all that keen on adding yet another set of > > trampolines, especially given we have constraints on how we can branch > > to them which render this not that useful in common configurations (e.g. > > where KASLR and module randomization is enabled). > > Yes, that makes kprobe jump optimization hard to implement on > RISC architecture in general. (x86 has 32bit offset jump instruction) > To solve this issue, something like "intermedate jump area" is needed > for each module. (Or, overwriting multiple instructions) > > > > > So importantly, do we actually need this? I don't think the sampel is > > that compelling since we can already use ftrace to measure function > > latencies. > > That depends on what you use it for, as you may know, kprobes allows > you to put the probes on function body (and inlined function), > on the other hand, ftrace can put only on the entry of the function. > I guess Qi may want to use it for improving performance of BPF. > > (BTW, as far as I know, Jisheng Zhang once tried to implement > kprobe on ftrace, that may be more helpful in this example. > https://lore.kernel.org/linux-arm-kernel/20191225172625.69811b3e@xhacker.debian/T/#m23a7aa55d32d140ee6a92102534446cfd4a43007 > I will pick them up again) > > > > > If we do need this, I think we need to do some more substantial rework > > to address those branch range limitations. I know that we could permit > > arbitrary branching if we expand the ftrace-with-regs callsites to ~6 > > instructions, but that interacts rather poorly with stacktracing and > > will make the kernel a bit bigger. > > Would you mean we reuse the ftrace-with-regs callsites for kprobes? > > arm32 avoids this limitation partially with reserved text pages > for trampoline in the kernel. But I think that is also a partial > solution. It may not work with module randomization at least on > arm64. > > On arm64, I think there are several way to solve it. > > - Add optprobe trampoline buffer for each module. > This is the simplest way to solve this issue, but requires some > pages to be added to each module (and kernel). > > - Add intermediate trampoline area for each module. (2-stage jump) > This jumps into an intermediate trampoline entry, save a partial > registers and jump the actual trampoline using that register. > This can reduce the size of trampoline buffer for each module. > > - Replace multiple instructions with the above intermediate jump > code. (single jump, but replace multiple instructions) > This requires to emulate multiple instructions and also the > kprobe must decode the instructions in the target function to > identify the replaced instructions are in one basic block. But > no need to add intermediate trampoline area (page). > > > > > > > On Wed, Aug 18, 2021 at 03:33:36PM +0800, Qi Liu wrote: > > > This patch introduce optprobe for ARM64. In optprobe, probed > > > instruction is replaced by a branch instruction to detour > > > buffer. Detour buffer contains trampoline code and a call to > > > optimized_callback(). optimized_callback() calls opt_pre_handler() > > > to execute kprobe handler. > > > > > > Performance of optprobe on Hip08 platform is test using kprobe > > > example module[1] to analyze the latency of a kernel function, > > > and here is the result: > > > > > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/kprobes/kretprobe_example.c > > > > > > kprobe before optimized: > > > [280709.846380] do_empty returned 0 and took 1530 ns to execute > > > [280709.852057] do_empty returned 0 and took 550 ns to execute > > > [280709.857631] do_empty returned 0 and took 440 ns to execute > > > [280709.863215] do_empty returned 0 and took 380 ns to execute > > > [280709.868787] do_empty returned 0 and took 360 ns to execute > > > [280709.874362] do_empty returned 0 and took 340 ns to execute > > > [280709.879936] do_empty returned 0 and took 320 ns to execute > > > [280709.885505] do_empty returned 0 and took 300 ns to execute > > > [280709.891075] do_empty returned 0 and took 280 ns to execute > > > [280709.896646] do_empty returned 0 and took 290 ns to execute > > > [280709.902220] do_empty returned 0 and took 290 ns to execute > > > [280709.907807] do_empty returned 0 and took 290 ns to execute > > > > > > optprobe: > > > [ 2965.964572] do_empty returned 0 and took 90 ns to execute > > > [ 2965.969952] do_empty returned 0 and took 80 ns to execute > > > [ 2965.975332] do_empty returned 0 and took 70 ns to execute > > > [ 2965.980714] do_empty returned 0 and took 60 ns to execute > > > [ 2965.986128] do_empty returned 0 and took 80 ns to execute > > > [ 2965.991507] do_empty returned 0 and took 70 ns to execute > > > [ 2965.996884] do_empty returned 0 and took 70 ns to execute > > > [ 2966.002262] do_empty returned 0 and took 80 ns to execute > > > [ 2966.007642] do_empty returned 0 and took 70 ns to execute > > > [ 2966.013020] do_empty returned 0 and took 70 ns to execute > > > [ 2966.018400] do_empty returned 0 and took 70 ns to execute > > > [ 2966.023779] do_empty returned 0 and took 70 ns to execute > > > [ 2966.029158] do_empty returned 0 and took 70 ns to execute > > > > Do we have any examples of where this latency matters in practice? > > > > > > > > Signed-off-by: Qi Liu > > > > > > Note: > > > To guarantee the offset between probe point and kprobe pre_handler > > > is smaller than 128MiB, users should set > > > CONFIG_RANDOMIZE_MODULE_REGION_FULL=N or set nokaslr in command line, or > > > optprobe will not work and fall back to normal kprobe. > > > > Hmm... I don't think that's something we want to recommend, and > > certainly distros *should* use KASLR and > > CONFIG_RANDOMIZE_MODULE_REGION_FULL. > > > > What happens with defconfig? Do we always get the fallback behaviour? > > Yes, in such case, it fails back to normal kprobe. just one minor comment. as Qi pointed out before, bootargs nokaslr will make kernel built by defconfig use optprobe: nokaslr [KNL] When CONFIG_RANDOMIZE_BASE is set, this disables kernel and module base offset ASLR (Address Space Layout Randomization). in lab, while security is not a concern as online, it would be a good option. > Anyway, optprobe is a background optimization. User can not specify > which kprobe is optimized. That is automatically done. > > Thank you, > > > -- > Masami Hiramatsu Thanks barry _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel