From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A63B6C433B4 for ; Thu, 29 Apr 2021 10:48:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B35161418 for ; Thu, 29 Apr 2021 10:48:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240573AbhD2KtJ (ORCPT ); Thu, 29 Apr 2021 06:49:09 -0400 Received: from foss.arm.com ([217.140.110.172]:46806 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233114AbhD2KtI (ORCPT ); Thu, 29 Apr 2021 06:49:08 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C25001FB; Thu, 29 Apr 2021 03:48:20 -0700 (PDT) Received: from C02TD0UTHF1T.local (unknown [10.57.1.102]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AA9A13F70D; Thu, 29 Apr 2021 03:48:18 -0700 (PDT) Date: Thu, 29 Apr 2021 11:48:13 +0100 From: Mark Rutland To: Leo Yan Cc: Catalin Marinas , Will Deacon , Mark Brown , Miroslav Benes , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Masami Hiramatsu , "Madhavan T. Venkataraman" Subject: Re: [PATCH] arm64: stacktrace: Stop unwinding when the PC is zero Message-ID: <20210429104813.GA33550@C02TD0UTHF1T.local> References: <20210429014321.196606-1-leo.yan@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210429014321.196606-1-leo.yan@linaro.org> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Leo, On Thu, Apr 29, 2021 at 09:43:21AM +0800, Leo Yan wrote: > When use ftrace for stack trace, it reports the spurious frame with the > PC value is zero. This can be reproduced with commands: > > # cd /sys/kernel/debug/tracing/ > # echo "prev_pid == 0" > events/sched/sched_switch/filter > # echo stacktrace > events/sched/sched_switch/trigger > # echo 1 > events/sched/sched_switch/enable > # cat trace > > -0 [005] d..2 259.621390: sched_switch: ... > -0 [005] d..3 259.621394: > => __schedule > => schedule_idle > => do_idle > => cpu_startup_entry > => secondary_start_kernel > => 0 IIUC, this is my fault, and is an unintended side-effect of commit: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") ... since before prior to that, we'd implicitly create a terminal record in start_kernel and secondary_start_kernel by virtue of entering those functions with both FP and LR set to NULL. After that commit, we report the NULL LR before trying to unwind the NULL FP. > The kernel initializes FP/PC values as zero for swapper threads in > head.S, when walk the stack frame, this patch stops unwinding if detect > the PC value is zero, therefore can avoid the spurious frame. > > Below is the stacktrace after applying the change: > > # cat trace > > -0 [005] d..2 259.621390: sched_switch: ... > -0 [005] d..3 259.621394: > => __schedule > => schedule_idle > => do_idle > => cpu_startup_entry > => secondary_start_kernel > > Signed-off-by: Leo Yan > --- > arch/arm64/kernel/stacktrace.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c > index 84b676bcf867..02b1e85b2026 100644 > --- a/arch/arm64/kernel/stacktrace.c > +++ b/arch/arm64/kernel/stacktrace.c > @@ -145,7 +145,11 @@ void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame, > if (!fn(data, frame->pc)) > break; > ret = unwind_frame(tsk, frame); > - if (ret < 0) > + /* > + * When the frame->pc is zero, it has reached to the initial pc > + * and fp values; stop unwinding for this case. > + */ > + if (ret < 0 || !frame->pc) > break; I don't think this is the right place for this, since we intend unwind_frame() to detect when unwinding is finished; see commit: 3c02600144bdb0a1 ("arm64: stacktrace: Report when we reach the end of the stack") I think we have three options for what to do here: a) Revert 6106e1112cc69a36, and identify these cases as terminal records where FP and LR are both NULL. b) Have __primary_switched and __secondary_switched call start_kernel and secondary_start_kernel with BL rather than B. The __*_switched functions will show up in the trace, but we won't unwind any further as the next record will have a NULL FP. c) Revert 6106e1112cc69a36, create terminal records in __primary_switched and __secondary_switched, and call start_kernel and secondary_start_kernel with BL rather than B. The __*_switched functions will show up in the trace, but we won't unwind any further as the next record will be a terminal record. For RELIABLE_STACKTRACE, we're going to have to do (c), I think, but for now we could do (a) so as to have a minimal fix, and we can build (c) atop that. How about the patch below? I've tested it with your instructions and also by inspecting /proc/self/stack. Thanks, Mark. ---->8---- >From b99e647b34b74059f3013c09f12fbd542c7679fd Mon Sep 17 00:00:00 2001 From: Mark Rutland Date: Thu, 29 Apr 2021 11:20:04 +0100 Subject: [PATCH] arm64: stacktrace: restore terminal records We removed the terminal frame records in commit: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") ... on the assumption that as we no longer used them to find the pt_regs at exception boundaries, they were no longer necessary. However, Leo reports that as an unintended side-effect, this causes traces which cross secondary_start_kernel to terminate one entry too late, with a spurious "0" entry. There are a few ways we could sovle this, but as we're planning to use terminal records for RELIABLE_STACKTRACE, let's revert the logic change for now, keeping the update comments and accounting for the changes in commit: 3c02600144bdb0a1 ("arm64: stacktrace: Report when we reach the end of the stack") This is effectively a partial revert of commit: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") Signed-off-by: Mark Rutland Fixes: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") Reported-by: Leo Yan Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Brown Cc: "Madhavan T. Venkataraman" --- arch/arm64/kernel/entry.S | 6 +++--- arch/arm64/kernel/stacktrace.c | 10 ++++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 6acfc5e6b5e0..9b205744a233 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -263,16 +263,16 @@ alternative_else_nop_endif stp lr, x21, [sp, #S_LR] /* - * For exceptions from EL0, terminate the callchain here. + * For exceptions from EL0, create a terminal frame record. * For exceptions from EL1, create a synthetic frame record so the * interrupted code shows up in the backtrace. */ .if \el == 0 - mov x29, xzr + stp xzr, xzr, [sp, #S_STACKFRAME] .else stp x29, x22, [sp, #S_STACKFRAME] - add x29, sp, #S_STACKFRAME .endif + add x29, sp, #S_STACKFRAME #ifdef CONFIG_ARM64_SW_TTBR0_PAN alternative_if_not ARM64_HAS_PAN diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index d55bdfb7789c..7032a5f9e624 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -44,10 +44,6 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) unsigned long fp = frame->fp; struct stack_info info; - /* Terminal record; nothing to unwind */ - if (!fp) - return -ENOENT; - if (fp & 0xf) return -EINVAL; @@ -108,6 +104,12 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) frame->pc = ptrauth_strip_insn_pac(frame->pc); + /* + * This is a terminal record, so we have finished unwinding. + */ + if (!frame->fp && !frame->pc) + return -ENOENT; + return 0; } NOKPROBE_SYMBOL(unwind_frame); -- 2.11.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B641C433ED for ; Thu, 29 Apr 2021 10:50:11 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BB7FD61409 for ; Thu, 29 Apr 2021 10:50:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BB7FD61409 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1778+f+4q4Q+ldvEcFbGQCNivN4cDWLEEtncPOJtVfo=; b=PfdjRqFFB+ckOGZzfQrX3Gysv MDjtwoh2kPDmyBecaoidWqrNI9j4QOUlvxKNZI8e0vBg458TM/HQ9h5zhr6oUvAOlALvWqx7S7Vou jfSoJgPubsDsl5zeus2q5ZjV7ax6W9DybtSC7oeBdC1oDrdyAx52Gc3MLq/fDsbTPU5B8V7BUQ9pX Y64wQLDXuj/WYQWYyTk06smrDMo0mbSAQL2/+DDhry85IWik+hdtvOJsTTblkumhXOvKPeO0sQML2 DQWlDB2/rjD2tJYSGQYH7aI5pBzsUwfnOKGtf+TQsJTf2TozRoiqb1XGXF49QkbSbV0Kk/VvREbj4 E8uk28FTw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lc4Df-005VqL-2z; Thu, 29 Apr 2021 10:48:31 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lc4Dc-005VqB-RK for linux-arm-kernel@desiato.infradead.org; Thu, 29 Apr 2021 10:48:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=In-Reply-To:Content-Type:MIME-Version :References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=xkfxMkDR921pfjZrvgFy1YvLaUJPbamjN4NOC36FAKE=; b=oJpge+zXOrE6CggFtt1EbKm3WY P0cDaay6tYqpOeADwI56+QODT9JMHrdOsY3+IkVjjndtAgWFNPuAn0XvGp68ifm5/VyCXD7gIb7fr K3fr6hC3aIDyk0fFoH61l+XYdXmQ9xQhDHAAzcL5fAPHak+DeJqsICaPR2k1O0XQiKrmDjY6yH01I VEokj+Sk5UiVKldM/m6ceK9huxKe+Nxp/iS2uGrTMpVG2Aav4zmjXz7r0mW53ebMat6cHSNaFGe/A JGop4oS+K/M5r2db7EemPcXrL9VchRNEZr8EWxwEhUsk8rbV0CQ36WJEv+/V40/asWwhGZEhFAlP0 UMSlF+Qg==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lc4DZ-000ZLt-F0 for linux-arm-kernel@lists.infradead.org; Thu, 29 Apr 2021 10:48:27 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C25001FB; Thu, 29 Apr 2021 03:48:20 -0700 (PDT) Received: from C02TD0UTHF1T.local (unknown [10.57.1.102]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AA9A13F70D; Thu, 29 Apr 2021 03:48:18 -0700 (PDT) Date: Thu, 29 Apr 2021 11:48:13 +0100 From: Mark Rutland To: Leo Yan Cc: Catalin Marinas , Will Deacon , Mark Brown , Miroslav Benes , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Masami Hiramatsu , "Madhavan T. Venkataraman" Subject: Re: [PATCH] arm64: stacktrace: Stop unwinding when the PC is zero Message-ID: <20210429104813.GA33550@C02TD0UTHF1T.local> References: <20210429014321.196606-1-leo.yan@linaro.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210429014321.196606-1-leo.yan@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210429_034825_661138_F4CF14CA X-CRM114-Status: GOOD ( 39.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Leo, On Thu, Apr 29, 2021 at 09:43:21AM +0800, Leo Yan wrote: > When use ftrace for stack trace, it reports the spurious frame with the > PC value is zero. This can be reproduced with commands: > > # cd /sys/kernel/debug/tracing/ > # echo "prev_pid == 0" > events/sched/sched_switch/filter > # echo stacktrace > events/sched/sched_switch/trigger > # echo 1 > events/sched/sched_switch/enable > # cat trace > > -0 [005] d..2 259.621390: sched_switch: ... > -0 [005] d..3 259.621394: > => __schedule > => schedule_idle > => do_idle > => cpu_startup_entry > => secondary_start_kernel > => 0 IIUC, this is my fault, and is an unintended side-effect of commit: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") ... since before prior to that, we'd implicitly create a terminal record in start_kernel and secondary_start_kernel by virtue of entering those functions with both FP and LR set to NULL. After that commit, we report the NULL LR before trying to unwind the NULL FP. > The kernel initializes FP/PC values as zero for swapper threads in > head.S, when walk the stack frame, this patch stops unwinding if detect > the PC value is zero, therefore can avoid the spurious frame. > > Below is the stacktrace after applying the change: > > # cat trace > > -0 [005] d..2 259.621390: sched_switch: ... > -0 [005] d..3 259.621394: > => __schedule > => schedule_idle > => do_idle > => cpu_startup_entry > => secondary_start_kernel > > Signed-off-by: Leo Yan > --- > arch/arm64/kernel/stacktrace.c | 6 +++++- > 1 file changed, 5 insertions(+), 1 deletion(-) > > diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c > index 84b676bcf867..02b1e85b2026 100644 > --- a/arch/arm64/kernel/stacktrace.c > +++ b/arch/arm64/kernel/stacktrace.c > @@ -145,7 +145,11 @@ void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame, > if (!fn(data, frame->pc)) > break; > ret = unwind_frame(tsk, frame); > - if (ret < 0) > + /* > + * When the frame->pc is zero, it has reached to the initial pc > + * and fp values; stop unwinding for this case. > + */ > + if (ret < 0 || !frame->pc) > break; I don't think this is the right place for this, since we intend unwind_frame() to detect when unwinding is finished; see commit: 3c02600144bdb0a1 ("arm64: stacktrace: Report when we reach the end of the stack") I think we have three options for what to do here: a) Revert 6106e1112cc69a36, and identify these cases as terminal records where FP and LR are both NULL. b) Have __primary_switched and __secondary_switched call start_kernel and secondary_start_kernel with BL rather than B. The __*_switched functions will show up in the trace, but we won't unwind any further as the next record will have a NULL FP. c) Revert 6106e1112cc69a36, create terminal records in __primary_switched and __secondary_switched, and call start_kernel and secondary_start_kernel with BL rather than B. The __*_switched functions will show up in the trace, but we won't unwind any further as the next record will be a terminal record. For RELIABLE_STACKTRACE, we're going to have to do (c), I think, but for now we could do (a) so as to have a minimal fix, and we can build (c) atop that. How about the patch below? I've tested it with your instructions and also by inspecting /proc/self/stack. Thanks, Mark. ---->8---- >From b99e647b34b74059f3013c09f12fbd542c7679fd Mon Sep 17 00:00:00 2001 From: Mark Rutland Date: Thu, 29 Apr 2021 11:20:04 +0100 Subject: [PATCH] arm64: stacktrace: restore terminal records We removed the terminal frame records in commit: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") ... on the assumption that as we no longer used them to find the pt_regs at exception boundaries, they were no longer necessary. However, Leo reports that as an unintended side-effect, this causes traces which cross secondary_start_kernel to terminate one entry too late, with a spurious "0" entry. There are a few ways we could sovle this, but as we're planning to use terminal records for RELIABLE_STACKTRACE, let's revert the logic change for now, keeping the update comments and accounting for the changes in commit: 3c02600144bdb0a1 ("arm64: stacktrace: Report when we reach the end of the stack") This is effectively a partial revert of commit: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") Signed-off-by: Mark Rutland Fixes: 6106e1112cc69a36 ("arm64: remove EL0 exception frame record") Reported-by: Leo Yan Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Brown Cc: "Madhavan T. Venkataraman" --- arch/arm64/kernel/entry.S | 6 +++--- arch/arm64/kernel/stacktrace.c | 10 ++++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 6acfc5e6b5e0..9b205744a233 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -263,16 +263,16 @@ alternative_else_nop_endif stp lr, x21, [sp, #S_LR] /* - * For exceptions from EL0, terminate the callchain here. + * For exceptions from EL0, create a terminal frame record. * For exceptions from EL1, create a synthetic frame record so the * interrupted code shows up in the backtrace. */ .if \el == 0 - mov x29, xzr + stp xzr, xzr, [sp, #S_STACKFRAME] .else stp x29, x22, [sp, #S_STACKFRAME] - add x29, sp, #S_STACKFRAME .endif + add x29, sp, #S_STACKFRAME #ifdef CONFIG_ARM64_SW_TTBR0_PAN alternative_if_not ARM64_HAS_PAN diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index d55bdfb7789c..7032a5f9e624 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -44,10 +44,6 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) unsigned long fp = frame->fp; struct stack_info info; - /* Terminal record; nothing to unwind */ - if (!fp) - return -ENOENT; - if (fp & 0xf) return -EINVAL; @@ -108,6 +104,12 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame) frame->pc = ptrauth_strip_insn_pac(frame->pc); + /* + * This is a terminal record, so we have finished unwinding. + */ + if (!frame->fp && !frame->pc) + return -ENOENT; + return 0; } NOKPROBE_SYMBOL(unwind_frame); -- 2.11.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel