From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 287E7C4338F for ; Tue, 24 Aug 2021 11:52:46 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DB03361246 for ; Tue, 24 Aug 2021 11:52:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DB03361246 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iPTpIUipams1YMO86EMX4pzR06DKVxWYL2qKLoj74ho=; b=jV9rin0AdewWvX C75JeAzoqUZbO6r0pY+aLMHL4Ki6x2UMm8NSUHbFRPdOPuCYervWyQ9c/+EzsNdddX+qJX+b9buxV 30FOY0S3xIuBDHlR5hKbrfgEN45Q9+CTsa33crJoisiVqAGO9LHzMuWn52+XneM57rZO2qQXx3CvC ZjtUPVsOj3QaH9OYV+kfCXn0o7TS7BAfyx74CrsrM70I+a93oSlKJ9e+WDAbgUMnL55xKE8jGWniQ 1ADdMUcZQjfjaIjpHwyVPepm+AzC38f/GcchNo0SJRaI7CLE8Xfef+FzgHKN2IkF1ZkKJGaAc097p xhGJ7hcAAUPX4VNkRo/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mIUwm-002qiL-NC; Tue, 24 Aug 2021 11:50:28 +0000 Received: from mail-ej1-x62b.google.com ([2a00:1450:4864:20::62b]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mIUwg-002qgl-CW for linux-arm-kernel@lists.infradead.org; Tue, 24 Aug 2021 11:50:26 +0000 Received: by mail-ej1-x62b.google.com with SMTP id h9so43790940ejs.4 for ; Tue, 24 Aug 2021 04:50:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=aPOK4hYPEQlsB4E9tsLxa9pHfmjZWvkYyLNBn0YcO2M=; b=ANrWX1O1fiBUlp7vTv0SjcNHNJYHsrGDKX2nHz9HPI+mW0SUSLhg/3M/6CQIgNEqEV PthBsHNj/meod3+I+pEJspyjwdAFD6Px5Ke98Rr2B8SULvRNWMRe8RWdJFjs1u6L9guL tLJKo7y6LDM8gimyDou0hrce72N7PM2CKjkv8NT8aelFfOLoA75sUR/aRcVnr7WxRVps wvNYWayXVot5fPpyf+nUAiq3BJ21YM9+llQavpdFKUNTgkwiVfTMWhZFQol+lFewEbub IDyafc80H9HbSVBZoWSECcx16Br5c2RCO8sawtVHdPP4af6VzkQ6LOkc8ePoHyoOZQB9 vQDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=aPOK4hYPEQlsB4E9tsLxa9pHfmjZWvkYyLNBn0YcO2M=; b=ahnNlkKuwu294hFA7UD7j+5UMZCQN4wiFSL6JcSfoE2lnev4Zp5EE6p7VOMhXe5+/H j99kO+uRcmIm73f3g1qeuimhDe2Z7JxwZoaIlzN1loM0P48AqKs9bCH3FkcmjckI2bDb BXxMFanKk3Wu4KLmH5gJYOPcE6FaVpoQkIC70C57lntBgm8gwk7kZEfpSPeqiLXt5sk/ dfBSvQ11X1+Pz5hMH8BGcpM1p2GJFa2VyLimGbrefJ2yoWUukWRMuZpD4kzA+6ZPwiyO 4iqqOBcVFQvFeHjnmawy5jG90E1m6Af3woJVnPEb8kcsY8OOD6vD0Ax25iJCnmJ+IfON vIzQ== X-Gm-Message-State: AOAM531o7UEyxjBtlVMzNgumnpWj2/3AeueJYluOqAYx7ce6gNlB9NAj wTm3eUyCrwHZ4SSJENCdKVTVaHtdgpmHpwMJF0s= X-Google-Smtp-Source: ABdhPJxDOAn/6jn2D8P+8YwI/N8jLQHyHoO2fDsuGZWAcD3ItiglJUaoR+mKmes4lcdc2wgAvcjFfU9D3UFrObqU1dY= X-Received: by 2002:a17:906:5a66:: with SMTP id my38mr10884140ejc.36.1629805819958; Tue, 24 Aug 2021 04:50:19 -0700 (PDT) MIME-Version: 1.0 References: <20210818073336.59678-1-liuqi115@huawei.com> <20210818073336.59678-3-liuqi115@huawei.com> <20210824105001.GA96738@C02TD0UTHF1T.local> In-Reply-To: <20210824105001.GA96738@C02TD0UTHF1T.local> From: Barry Song <21cnbao@gmail.com> Date: Tue, 24 Aug 2021 23:50:08 +1200 Message-ID: Subject: Re: [PATCH v4 2/2] arm64: kprobe: Enable OPTPROBE for arm64 To: Mark Rutland Cc: Qi Liu , Catalin Marinas , will@kernel.org, naveen.n.rao@linux.ibm.com, anil.s.keshavamurthy@intel.com, David Miller , mhiramat@kernel.org, linux-arm-kernel@lists.infradead.org, Barry Song , prime.zeng@hisilicon.com, robin.murphy@arm.com, f.fangjian@huawei.com, Linuxarm , LKML X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210824_045022_492459_57CD52A4 X-CRM114-Status: GOOD ( 36.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Aug 24, 2021 at 10:51 PM Mark Rutland wrote: > > Hi, > > I have a bunch of comments below. > > At a high-level, I'm not all that keen on adding yet another set of > trampolines, especially given we have constraints on how we can branch > to them which render this not that useful in common configurations (e.g. > where KASLR and module randomization is enabled). > > So importantly, do we actually need this? I don't think the sampel is > that compelling since we can already use ftrace to measure function > latencies. Hopefully I can help answer some general questions. for code details, I'll still rely on liuqi. As far as I know, the functionality of ftrace is very limited. kprobe, on the other hand, can do a lot of things, for example, hooking some ebpf code. Tons of userspace tools depend on kprobe, eg. ebpf/bcc which is widely used nowadays: https://github.com/iovisor/bcc Function latency is really a very small part of what kprobe can do. kprobe can do much much more. For example, while doing disksnoop by kprobe, we are easily collecting the request struct, and get the layout of each IO request. without kprobe, we will probably need to add tracepoints here and there in kernel code to achieve the same goal. I believe Masami and kprobe maintainers can answer this question better than me. > > If we do need this, I think we need to do some more substantial rework > to address those branch range limitations. I know that we could permit > arbitrary branching if we expand the ftrace-with-regs callsites to ~6 > instructions, but that interacts rather poorly with stacktracing and > will make the kernel a bit bigger. This patch is probably the first step towards a faster kprobe on ARM64. I assume PLT or some similar tech will help break the limitation next step. > > On Wed, Aug 18, 2021 at 03:33:36PM +0800, Qi Liu wrote: > > This patch introduce optprobe for ARM64. In optprobe, probed > > instruction is replaced by a branch instruction to detour > > buffer. Detour buffer contains trampoline code and a call to > > optimized_callback(). optimized_callback() calls opt_pre_handler() > > to execute kprobe handler. > > > > Performance of optprobe on Hip08 platform is test using kprobe > > example module[1] to analyze the latency of a kernel function, > > and here is the result: > > > > [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/samples/kprobes/kretprobe_example.c > > > > kprobe before optimized: > > [280709.846380] do_empty returned 0 and took 1530 ns to execute > > [280709.852057] do_empty returned 0 and took 550 ns to execute > > [280709.857631] do_empty returned 0 and took 440 ns to execute > > [280709.863215] do_empty returned 0 and took 380 ns to execute > > [280709.868787] do_empty returned 0 and took 360 ns to execute > > [280709.874362] do_empty returned 0 and took 340 ns to execute > > [280709.879936] do_empty returned 0 and took 320 ns to execute > > [280709.885505] do_empty returned 0 and took 300 ns to execute > > [280709.891075] do_empty returned 0 and took 280 ns to execute > > [280709.896646] do_empty returned 0 and took 290 ns to execute > > [280709.902220] do_empty returned 0 and took 290 ns to execute > > [280709.907807] do_empty returned 0 and took 290 ns to execute > > > > optprobe: > > [ 2965.964572] do_empty returned 0 and took 90 ns to execute > > [ 2965.969952] do_empty returned 0 and took 80 ns to execute > > [ 2965.975332] do_empty returned 0 and took 70 ns to execute > > [ 2965.980714] do_empty returned 0 and took 60 ns to execute > > [ 2965.986128] do_empty returned 0 and took 80 ns to execute > > [ 2965.991507] do_empty returned 0 and took 70 ns to execute > > [ 2965.996884] do_empty returned 0 and took 70 ns to execute > > [ 2966.002262] do_empty returned 0 and took 80 ns to execute > > [ 2966.007642] do_empty returned 0 and took 70 ns to execute > > [ 2966.013020] do_empty returned 0 and took 70 ns to execute > > [ 2966.018400] do_empty returned 0 and took 70 ns to execute > > [ 2966.023779] do_empty returned 0 and took 70 ns to execute > > [ 2966.029158] do_empty returned 0 and took 70 ns to execute > > Do we have any examples of where this latency matters in practice? While doing performance analysis or debugging performance issues in lab, the big latency of the current kprobe can significantly impact the real result. And it is particularly important when we want to do some online diagnostics where machines are running on the cloud. we need to minimally affect the real system while watching the system. For example, https://gitee.com/xiebaoyou/diagnosis_tools is an open-source tool which is used by alibaba, it uses kprobe a lot to detect various problems running on aloud. > > > > > Signed-off-by: Qi Liu > > > > Note: > > To guarantee the offset between probe point and kprobe pre_handler > > is smaller than 128MiB, users should set > > CONFIG_RANDOMIZE_MODULE_REGION_FULL=N or set nokaslr in command line, or > > optprobe will not work and fall back to normal kprobe. > > Hmm... I don't think that's something we want to recommend, and > certainly distros *should* use KASLR and > CONFIG_RANDOMIZE_MODULE_REGION_FULL. > > What happens with defconfig? Do we always get the fallback behaviour? For this stage, with no_kaslr cmdline for defconfig, we are always going to optprobe. otherwise, it always goes to the fallback behaviour. So for this stage, it is probably only useful in lab while people are debugging problems. but this is still a widely useful scenario. > Thanks Barry _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel